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AFIT/GIR/LAL/96D-3 

Abstract 

This  study  investigated  user  requirements  for  an  on-line  system  to  distribute  United 
States  Air  Force  publications.  Requirements  were  gathered  utilizing  focus  groups  of  users 
from  the  students  and  staff  of  the  Air  Force  Institute  of  Technology.  The  focus  groups 
identified  requirements  for  the  system  intended  to  foster  its  acceptance  within  the  user 
community. 

The  conceptual  framework  posited  user  involvement  in  the  design  and 
implementation  of  the  new  system,  providing  a  balance  between  functionality  and  ease  of 
use  fostering  user  acceptance.  The  framework  was  used  to  examine  the  relationships 
between  the  users  involvement  in  the  system  design  and  implementation  and  its  acceptance 
and  use. 

The  conceptual  framework  and  focus  groups  were  used  for  gathering  data  about 
system  requirements.  The  inputs  from  the  focus  groups  were  used  to  develop  the 
suggested  requirements  for  the  system.  The  resulting  system  requirements  fell  into  the 
following  groups:  system  reliability,  system  accessibility,  search  capabilities,  desktop 
options,  and  hyperlink  capabilities. 
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ON-LINE  PUBLICATIONS: 


DEFINING  REQUIREMENTS  FOR  USER  ACCEPTANCE 

I.  Introduction 


Background 

The  United  States  Air  Force  distributes  Policy  Directives,  Instructions, 
Handbooks,  Catalogs  and  other  types  of  organizational  information  using  CD  ROM  and 
paper  copies.  These  methods  of  getting  the  information  to  the  field  units  require  an 
expensive  physical  distribution  network.  In  addition  to  the  physical  distribution  methods, 
there  are  unofficial  Internet  sites  that  provide  access  to  some  of  the  publications.  The  Air 
Force  strategic  plan  for  information  management  calls  for  the  migration  from  paper  to  a 
completely  electronic  form  of  information  distribution  (Air  Force,  1996). 

The  physical  distribution  of  organizational  information  is  expensive  and  wasteful. 

The  Air  Force  Electronic  Publishing  Master  Program  Guide  reports: 

There  is  an  Air  Force-leased  GSA  warehouse  in  Baltimore  that  contains  Air 
Force  publications  and  forms  destined  for  distribution  to  field  users.  That 
warehouse  is  bigger  than  six  football  fields.  Every  MAJCOM  has  a  smaller, 
but  similar,  facility.  Each  base  has  a  similar  warehouse  which  stores  and 
distributes  AF  and  MAJCOM  products  to  the  end  users.  We  can  nearly 
elintinate  these  facilities  if  we  were  to  put  every  possible  product  into  an 
electronic  version  that  Air  Force  people  can  access  on  their  personal 
computers.  This  will  save  in  reduced:  warehouse  requirements;  overhead  to 
handle  the  products;  and  postage  and  transportation  costs  to  move  them  from 
point  to  point.  While  we  cannot  eliminate  storage  and  transport  of  every 
published  item  because  some  products  are  not  easily  adaptable  for  the 
electronic  environment,  the  other  63  percent  is,  and  we  need  to  focus  on  that 
area.  (Air  Force,  1996:8) 

A  better  method  of  getting  this  information  to  those  who  need  it  in  the  Air  Force  (AF) 
could  save  a  significant  amount  of  money. 
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The  AF  plan  for  the  migration  toward  totally  electronic  publishing  calls  for 
maintaining  some  form  of  physical  distribution  system  for  a  period  of  time  into  the  fumre 
(Air  Force,  1996).  Some  documents  are  not  readily  transferable  to  a  completely  electronic 
format  as  they  require  carbon  copies,  are  in  a  specific  format  for  display,  or  other  similar 
reasons.  The  AF  strategic  publishing  plan  calls  for  focusing  on  the  publications  that  are 
easily  converted  first.  The  reduction  in  distribution  of  paper  copies  of  these  publications 
offer  significant  savings  for  the  AF  in  printing,  storage,  and  transportation  costs. 

While  this  monetary  savings  is  important,  productivity  gains  could  outweigh  them 
in  terms  of  value  to  the  Air  Force.  Today  the  employee  operating  in  the  field  looks  for 
information  in  the  cumbersome  volumes  of  paper  documentation.  They  first  search  the 
large  three-ring  binder  of  the  paper  table  of  contents,  known  as  the  “0-2,”  for  the  name 
and  number  of  the  publication  for  the  topic  of  interest.  Providing  they  find  what  they’re 
looking  for,  and  the  publications  library  contains  a  copy  of  the  particular  instruction  or 
other  guidance,  they  can  look  up  the  desired  topic  in  the  appropriate  binder.  The 
information  the  individual  finds  could  be  inaccurate  as  the  office  of  primary  responsibility 
(OPR)  may  have  sent  out  a  change  in  policy  with  an  interim  message  change  (IMG)  that 
has  not  been  received  or  is  not  yet  posted  to  the  library  volume. 

Producing  the  information  in  a  completely  electronic  format  could  provide  many 
opportunities  to  improve  the  previous  scenario.  The  individual  could  use  a  query 
capability  to  find  the  right  information.  The  information  would  be  more  reliable  as  any 
changes  the  OPR  would  make  could  be  made  immediately  available  to  the  viewer. 
Additionally,  the  information  discovered  often  references  other  publications,  and  with  the 
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electronic  format  of  an  on-line  publishing  system,  could  link  to  the  other  publication 
directly.  The  individual  would  have  almost  instant  access  to  the  referenced  document  with 
this  hypertext  ability.  The  ability  to  hyperlink  also  exists  in  an  on-line  system.  This  would 
allow  the  text  to  reference  other  related  text,  a  picture,  sound  clip,  video,  or  other  related 
material  to  help  the  individual  understand  the  guidance  quicker  and  more  completely.  The 
gains  in  increased  productivity  are  difficult  to  measure  in  monetary  terms,  but  could  have  a 
dramatic  affect  on  the  way  we  do  business  in  the  United  States  Air  Force. 

The  current  standard  for  the  AF  is  to  utilize  personal  computers  in  most  work 
stations  around  the  organization.  The  software  programs  the  AF  has  standardized  are 
Windows  compatible  graphical  interface-type  applications.  The  average  user  would 
benefit  from  the  proposed  publications  system  being  interoperable  with  the  Windows 
compatible  environment. 

Specific  Purpose 

The  long-term  solution  planned  for  the  future  of  AF  publication  is  a  completely 
electronic  information  system  from  authoring,  to  coordination  and  approval,  to 
supplementation  by  lower  headquarters  and  use  (Air  Force,  1996).  One  of  the  key  steps  in 
developing  a  system  like  this  proposed  on-line  publications  system  is  determining  the 
requirements.  Deciding  what  the  system  is  supposed  to  do  and  how  it  will  accomplish  its 
objectives  in  the  early  phases  of  the  project  often  determines  the  success  or  failure  of  the 
venture. 
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Research  Question 


The  purpose  of  this  study  is  to  determine  what  the  users  require  from  an  electronic 
publishing  system  in  order  to  accept  and  use  the  system  for  day  to  day  operations. 
Authoring,  storing,  and  using  information  only  on  the  computer  is  a  shift  from  the 
paradigm  of  a  fully  paper-based  system.  What  can  an  on-line,  electronic  publishing  system 
offer  users  to  make  this  change  easier  to  deal  with? 

Investigative  Questions 

In  order  to  answer  the  research  question,  the  following  investigative  questions  are 
answered  in  this  thesis. 

1.  What  functions  should  the  system  offer  the  user  in  terms  of  the  work 
environment  and/or  the  desktop  environment? 

2.  On  which  types  of  items  should  the  user  be  able  to  query  the  publication 
system:  i.e.  key  terms,  subject  areas  or  other  ideas? 

3.  What  should  the  graphical  interface  for  the  on-line  publications  system  look 
like? 

To  provide  preliminary  answers  to  these  questions  a  qualitative  research  approach  relying 
on  focus  groups  of  potential  users  was  used. 

Research  Layout 

The  literature  review  in  chapter  II  presents  some  underlying  constructs  in  the 
relevant  disciplines  used  to  develop  the  conceptual  framework  behind  determining  user 
requirements  for  the  system.  Chapter  HI  discusses  the  methodology  used  in  setting  up  and 
performing  the  focus  groups.  Chapter  IV  interprets  the  data  gathered  by  the  focus  groups 
to  determine  its  usefulness  for  the  stated  purpose.  The  analysis  of  the  data  will  show 
what  functions  the  users  feel  are  the  most  important  and  the  types  of  items  on  which  the 
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users  would  like  to  query  the  system.  Chapter  V  will  make  conclusions  and 
recommendations  concerning  the  requirements  for  the  on-line  publishing  system  and  for 
future  areas  of  research. 

Summary 

This  chapter  outlined  the  need  for  and  the  plan  of  the  United  States  Air  Force  to 
make  its  published  information  available  on-line.  An  investigation  into  the  user’s 
requirements  for  such  a  system  was  proposed  to  help  ensure  the  users  would  accept  and 
use  the  new  method  of  publishing  information. 
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II.  Literature  Review 


Introduction 

This  literature  review  covers  the  topics  of  systems  development,  requirements 
analysis,  and  the  constmcts  contained  in  the  user  acceptance  conceptual  model  to  be 
introduced  in  Chapter  III.  The  four  constructs  to  be  examined  concerning  the  conceptual 
framework  for  this  study  are  user  satisfaction,  ease  of  use,  functionality,  and  user 
involvement.  The  model  addresses  the  relationship  of  these  constmcts  to  the  acceptance 
of  a  new  information  system  by  the  user. 

Electronic  Publishing 

The  word  publish  means  many  things  to  many  people,  but  most  sources  agree  to 
the  basic  idea  that  it  means  to  make  public  (Doty  and  Bishop,  1994;  Peek,  1994).  The 
common  understanding  includes  producing  a  printed  copy  for  use  by  others.  This 
understanding  is  evolving  rapidly  with  electronic  publishing  and  the  advances  in  computers 
and  networks.  The  possibilities  and  the  challenges  presented  by  the  new  technology  are 
great. 

Information  is  the  key  source  of  wealth  in  the  postindustrial  age  (Jacobson,  1994). 
Many  companies  are  beginning  to  recognize  that  electronic  publishing  is  an  enterprise- 
wide  activity  to  get  information  to  those  who  need  it  (Francis,  1990).  Several  questions 
need  to  be  answered  before  the  electronic  publishing  system  is  designed.  An  organization 
must  determine  who  needs  to  see  the  information,  what  level  of  the  corporate  data  to 
which  they  should  have  access,  whether  they  should  be  able  to  change  the  data,  or  who 
else  needs  to  see  the  data  (Beal,  1991).  Clifford  Lynch  (1994)  points  out  the  need  to 
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consider  the  security  and  integrity  of  the  electronic  information.  Users  need  to  be  sure  of 
who  authored  the  information  and  that  it  was  not  altered  after  it  was  written  and  made 
available. 

Steps  to  Electronic  Publishing 

The  first  step  to  electronic  publishing  involves  moving  the  current  paper 
documents  to  digital  images  on  the  computer.  The  original  answer  was  to  scan  the  image 
and  make  a  picture  of  it  in  the  computer.  The  problem  with  this  method,  according  to 
Cary  Lu  (1993),  is  that  the  storage  consists  only  of  images  and  can’t  be  read  as  text  by  the 
computer.  Optical  character  recognition  (OCR)  was  developed  to  solve  this  problem. 
OCR  uses  a  scanning  process  that  includes  computer  software  to  translate  text  into  a 
computer-readable  format  OCR  does  not  work  perfectly  for  all  documents,  especially 
those  heavily  formatted  and  those  including  graphics. 

The  next  step  in  the  move  to  electronic  publishing  is  to  establish  a  system  to  index 
your  digitized  paper  and  digital  documents  for  search  and  retrieval  (Lu,  1993).  The  ability 
of  the  computer  to  read  the  document  as  text  is  instrumental  in  most  applications  of 
electronic  publishing.  The  document  could  still  be  read,  but  the  user  couldn’t  search  for  a 
particular  topic  or  piece  of  information.  There  are  many  uses  for  each  type  of  retrieval, 
and  the  use  of  the  document  will  drive  the  method  needed. 

An  important  consideration,  when  developing  an  electronic  publishing  strategy,  is 
ensuring  documents  can  be  used  with  various  platforms  and  applications.  The  computer 
industry  has  developed  standards  for  document  exchange  to  help  ensure  this  possibility. 
Documents  can  include  not  only  text,  but  audio,  graphics,  and  full  motion  video.  They 
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will  retain  the  same  look  and  feel  across  and  platform  or  application  with  the  established 
industry  standard  for  document  exchange. 

The  lack  of  an  industry  standard  for  indexing  electronic  documents  prevents  a  lot 
of  information  from  being  available  on-line.  Currently,  you  must  download  all  the 
documents  and  run  a  program  to  index  them,  in  order  to  query  them  for  the  information 
you  desire.  The  other  option  is  to  use  a  different  search  query  for  each  type  of  document 
database.  Without  a  standard  it  will  be  difficult  to  mix  information  from  outside  the 
organization  into  the  corporate  database.  Cary  Lu  (1993)  feels  it  will  be  some  time  before 
an  industry  standard  in  this  area  is  developed. 

The  current  literature  covers  many  of  the  theoretical  issues  surrounding  the  world 
of  electronic  publishing.  The  Air  Force  has  made  the  decision  to  migrate  toward  this 
technology.  The  initial  phase  of  the  transition  was  to  move  the  publications  onto  CD- 
ROM  and  distribute  them  to  all  the  units  around  the  world.  This  was  planned  as  an 
interim  step  on  the  way  to  fuUy  on-line  access  to  publications.  The  next  step  in  the 
migration,  was  to  decide  on  Standard  Generalized  Markup  Language  (SGML)  as  the 
format  for  creating  new  documents  and  for  converting  the  old  text  documents.  Jeffrey 
Hibbard  supports  this  choice  with  his  opinion  that  SGML  is  useful  for  manuals,  textbooks, 
technical  reports  and  articles,  proposals  and  other  similar  types  of  documents  (Hibbard, 
1990).  This  research  will  focus  on  the  requirements  the  user  has  for  a  system  of 
publications  available  on-line 
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Systems  Development 

The  information  systems  life  cycle  (ISLC)  presents  the  concept  that  information 
systems  have  a  useful  life  span  (Martin,  1995).  The  construct  of  the  ISLC  views 
information  systems  development  as  a  four  phase  process.  The  first  phase  is  the  design  of 
the  system.  Upon  the  completion  of  the  design,  the  system  is  implemented  and  fine  tuned. 
The  third  phase  of  the  ISLC  is  the  systems  operation,  including  the  regular  maintenance  of 
the  system.  The  final  life  cycle  phase  is  the  system  obsolescence  or  the  death  of  the 
system.  The  life  cycle  points  out  the  continuing  need  for  new  systems  development 
projects. 

Kendall  and  Kendall  (1995)  assert  structured  analysis  and  design  provides  a 
systematic  approach  to  designing  and  building  quality  computer  systems.  The  most  widely 
used  methodology  for  incorporating  this  systematic  approach  to  system  development  is 
the  systems  development  life  cycle  (SDLC).  Kendall  and  Kendall  (1995)  provide  a  model 
of  the  SDLC  with  seven  steps.  They  caution  analysts  disagree  on  exactly  how  many 
phases  there  are  in  the  SDLC,  but  they  laud  its  organized  approach  to  systems 
development. 

Different  interpretations  of  the  SDLC  are  common  in  field  as  exemplified  by  the 
various  authors  models  (A wad,  1985;  Ostle,  1985,  Kendall  and  Kendall,  1995;  and  Martin, 
1995).  Though  the  different  models  differ  in  the  number  of  steps  as  pointed  out  by 
Kendall  and  Kendall  (1995),  the  organization  the  models  provide  to  the  design  process  is 
the  important  thing.  Figure  1  depicts  the  SDLC  as  the  twelve  step  process  proposed  by 
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Martin  (1995)  in  his  text  on  system  analysis  and  design.  Martin  (1995)  depicts  the  process 
as  three  major  nodules  including:  analysis,  design,  and  implementation. 

The  analyst  determines  if  the  current  system  has  problems  and  what  design 
remedies  are  appropriate  in  the  analysis  phase  which  includes  five  steps:  problem 
detection,  initial  investigation,  requirements  analysis,  system  selection,  and  preliminary 
system  design.  The  four  steps  in  the  design  phase,  the  phase  where  the  system  selected  in 
the  analysis  phase,  include:  output,  input,  files,  and  processes.  Implementation,  the  phase 
of  the  process  where  logical  design  specifications  are  translated  into  the  actual 
constmction  of  the  information  system,  includes  the  remaining  three  steps  of  the  process: 
programming/testing,  system  change-over,  and  system  evaluation.  Martin  (1995)  is 
careful  to  point  out  the  SDLC  is  not  a  completely  linear  process  as  Figure  1  might 
suggest.  The  analyst  may  return  from  any  step  to  any  of  the  previous  steps  in  the  process. 
Kendall  and  Kendall  (1995)  support  this  view  as  they  state  the  steps  of  the  SDLC  are 
sequential,  but  are  interrelated  and  often  are  accomplished  simtiltaneously. 

Requirements  Analysis 

One  of  the  earliest  stages  in  the  successful  development  of  an  effective  information 
system  (IS)  is  a  thorough  understanding  of  the  user’s  needs  (Byrd  et  al,  1992).  Defining 
user  needs  is  commonly  called  requirements  analysis.  Requirements  analysis  involves  end 
users  and  systems  analysts  interacting  in  an  effort  to  recognize  and 
specify  the  data  and  information  needed  to  develop  an  information  system  (Byrd  et  al, 
1992).  Zmud  (1983)  declares  requirements  analysis  continues  to  be  the  most  critical  and 
most  difficult  activity  normally  undertaken  in  information  systems  development. 
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Byrd  et  al  (1992)  list  five  steps  commonly  performed  by  the  analyst  in  determining 
requirements: 

1)  working  with  end  users  to  establish  an  understanding  of  organizational 
information  processing  needs 

2)  developing  IS  objectives 

3)  designing  and  evaluation  IS  alternatives 

4)  communicating  the  results  of  analyses  to  superiors,  other  analysts,  and  end 
users 

5)  performing  a  systems  audit  (Byrd  et  al,  1 992: 117) 
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Figure  1.  Stages  of  the  Systems  Development  Life  Cycle  (SDLC) 


The  process  appears  quite  clear;  find  out  what  the  user  wants,  needs,  and  desires;  then 
use  this  information  to  provide  the  appropriate  computer  system. 
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Jeffrey  and  Putman  (1994)  discuss  two  concepts  that  prevent  the  activity  from 
being  the  straight-forward  endeavor  which  it  initially  appears.  First,  defining  user 
requirements  is  a  complex  division  of  labor  between  the  technology-oriented  system 
developers  and  the  organization-oriented  users.  The  developers  don’t  understand  the 
organization  nearly  as  well  as  the  users  working  in  it  on  daily  basis,  and  the  users  know 
the  organization  with  little  knowledge  of  the  technology  available  to  meet  their  needs. 

The  second  point  Jeffrey  and  Putman  (1994)  discuss  as  a  hindrance  to  the  process  is  that 
requirements  are  not  established  as  a  list  to  be  discovered,  but  are  hard  to  define  with 
clarity  as  the  system  is  intended  to  affect  a  change  to  the  organization  that  is  already  very 
ambiguous.  The  difficult  part  of  the  process  comes  down  to  communication  between  the 
users  and  the  developers.  The  user  knows  what  they  want  the  system  to  do,  but  they 
don’t  understand  what  technology  is  available  to  perform  the  task.  They  must  convey 
their  desires  to  the  technician  in  a  manner  that  allows  the  developer  to  come  up  with  a 
feasible  solution. 

Martin  (1995)  provides  an  analysis  of  several  methods  of  obtaining  data  for  the 
requirements  gathering  phase.  The  methods  he  analyzes  are  interviews,  questionnaires, 
procedure  analysis,  and  document  survey.  Interviews  are  stmctured  one-on-one  sessions 
the  researcher  scripts  and  tightly  controls.  Questionnaires  ask  the  opinion  of  the  users 
about  ideas  the  researcher  determines  to  be  relevant.  Procedure  analysis  involves 
watching  the  user  in  the  natural  environment  to  determine  what  they  need  to  perform  their 
job.  Document  survey  entails  analyzing  the  flow  of  documents  through  the  user’s 
processes  to  determine  how  the  system  could  help  the  flow. 
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Morgan  (1988)  discusses  the  focus  group  as  an  another  method  for  gathering 
qualitative  data  like  system’s  requirements.  He  points  out  the  technique  is  a  combination 
of  the  personal  interview  and  procedure  analysis  or  participant  observation.  According  to 
Morgan,  the  main  advantage  of  the  focus  group  over  participant  observation  is  the  ability 
to  observe  a  large  amount  of  interaction  in  a  limited  period  of  time.  He  cautions  the  focus 
group  lacks  the  natural  setting  of  participant  observation  and  shouldn’t  be  used  if  this  is 
important.  The  main  advantage  of  focus  groups  over  personal  interviews,  according  to 
Morgan  (1988)  is  the  ability  to  observe  interaction  on  a  topic.  The  focus  group  offers  less 
control  by  the  researcher  of  the  data  gathering  than  the  stmctured  individual  interview. 
The  interaction  of  the  group  participants  takes  the  place  of  the  researcher  control. 
According  to  Morgan  (1988),  the  situation  that  takes  advantage  of  the  strengths  of  focus 
groups  over  personal  interviews  and  participant  observation  is  appropriate  for  using  focus 
groups.  The  next  section  discusses  how  the  dynamics  of  user  acceptance  of  the  system 
should  influence  the  requirements  analysis  phase  of  SDLC. 

User  Satisfaction/Acceptance 

Ives  et  al  (1983)  define  user  information  satisfaction  (UIS)  as  the  extent  to  which 
users  beheve  the  information  system  available  to  them  meets  their  information 
requirements.  There  has  been  much  attention  paid  to  the  concept  of  user  satisfaction  as  a 
measure  of  success  of  information  systems  and  the  productivity  of  the  system  users.  Ives 
et  al  (1983)  assert  UIS  is  a  perceptual  or  subjective  measure  of  system  success;  it  serves  as 
a  substitute  for  objective  determinants  of  information  system  effectiveness  which  are 
frequently  not  available.  According  to  Hendrickson  et  al  (1993),  an  underlying  tenet  of 
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information  system  success  is  the  decision  maker’s  willingness  to  adopt  and  utilize  these 
systems. 

There  has  been  significant  research  into  determining  user  satisfaction.  Bailey  and 
Pearson  (1983)  created  one  of  the  earliest  tools  for  measuring  computer  user  satisfaction, 
a  39  item  user  satisfaction  survey.  Ives  et  al  (1983)  and  Baroudi  and  Orlikowski  (1988) 
continued  their  work  and  refined  the  instrument  to  include  22  and  13  items,  respectively, 
which  they  asserted  still  effectively  measured  the  user  acceptance  constmct.  Baroudi  and 
Orlikowski  (1988)  define  three  factors  these  studies  are  built  around  including: 

1.  EDP  (electronic  data  processing)  Staff  and  Services.  This  factor  is  the 
respondents’  self  reported  assessment  of  the  attitude  and  responsiveness  of 
the  EDP  staff  as  well  as  the  quality  of  their  relationship  with  the  EDP  staff. 

2.  Information  Product.  This  factor  is  the  respondents’  self-reported 
assessment  of  the  quality  of  output  delivered  by  the  information  system. 

3.  Knowledge  and  Involvement.  This  factor  is  the  respondents’  self-reported 
assessment  of  the  quality  of  training  provided,  their  understanding  of  the 
system,  and  their  participation  in  its  development.  (Baroudi  and  Orlikowski, 
1988:48) 

These  early  measures  of  user  satisfaction  included  the  measure  of  the  EDP  staff  and 
services  as  the  majority  of  the  computing  was  performed  by  the  department. 

The  increase  of  personal  computing  in  the  1980s  and  1990s  has  shifted  the  concept 
of  user  satisfaction  away  from  the  EDP  staff  and  services.  Doll  and  Torkzadeh  (1988) 
investigated  the  measurement  of  the  user  satisfaction  constmct  in  relation  to  the  growing 
field  of  end-user  computing  (EUC).  They  created  a  12-item  instrament  to  measure  end 
user  computing  satisfaction  (EUCS)  based  on  five  components  of  user  satisfaction, 
including:  content,  accuracy,  format,  ease  of  use,  and  timeliness. 
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Etezadi-Amoli  and  Farhoomand  (1996)  have  also  developed  a  EUCS  measure. 

The  instrument  was  developed  around  six  elements  of  EUCS  including:  documentation, 
ease  of  use,  functionality  of  system,  quality  of  output,  support,  and  security.  The  research 
supported  the  idea  that  their  user  satisfaction  measure  was  useful  in  determining  user 
performance  (Etezadi-Amoli  and  Farhoomand,  1996). 

Davis  (1989),  Goodwin  (1993),  and  Hendrickson  et  al  (1993)  demonstrate  a  trend 
in  the  literature  arguing  perceived  usefulness  and  ease  of  use  as  the  two  main  determinants 
of  acceptance  for  an  information  system.  Perceived  usefulness,  or  functionality,  as  it  is 
referred  in  this  study  represents  the  idea  that  people  tend  to  use  or  not  use  an  application 
to  the  extent  they  believe  it  will  help  them  perform  their  job  better  (Davis,  1989).  Despite 
the  usefulness  of  a  system,  ease  of  use  hypothesizes  that  people  will  only  take  advantage 
of  the  application  if  the  benefits  outweigh  the  effort  of  using  it  (Davis,  1989).  These  two 
constmcts  are  important  elements  for  the  discussion  of  user  acceptance  of  information 
systems. 

Ease  of  Use 

Davis  (1989),  Goodwin  (1993),  and  Doll  and  Torkzadeh  (1988)  are  examples  of 
EUC-based  satisfaction  measures  that  assert  ease  of  use  is  the  most  important  element  of 
user  satisfaction.  Goodwin  (1993)  argues  ease  of  use,  or  how  something  is  accomplished, 
can  be  as  important  as  what  functions  are  available  in  an  application.  Ease  of  use  has 
become  more  important  as  the  technology  has  progressed,  making  it  an  important  element 
for  consideration  during  the  analysis  of  the  necessary  system  architecmre  (Branscomb  and 
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Thomas,  1984).  Users  with  powerful  personal  computers  expect  sophisticated  software  to 

perform  the  functions  they  need,  and  still  be  easy  to  use. 

The  established  user  satisfaction  instruments  consistently  use  ease  of  use  as  a 

major  element  of  the  construct.  Additionally,  several  works  from  the  literature  argue  ease 

of  use  is  the  most  important  element  of  user  satisfaction  (Branscomb  and  Thomas,  1984; 

Doll  and  Torkzadeh,  1988;  Goodwin,  1993). 

According  to  Davis  (1989),  ease  of  use  refers  to  the  degree  to  which  a  person 

believes  that  using  a  particular  system  would  be  free  from  effort.  Goodwin  (1993) 

describes  usability,  another  term  for  ease  of  use,  in  the  following  manner: 

It  is  affected  by  the  types  of  tasks  to  be  accomplished:  A  keyboard-based  interface 
appropriate  for  a  word-processing  application  may  be  inadequate  for  a  graphics 
application.  In  this  respect,  usability,  like  functionality,  is  task  related;  it  is  also 
people  related.  The  characteristics  that  make  a  system  usable  for  one  set  of  users 
may  render  it  unusable  for  another.  First-time,  casual,  and  expert  users  may  all 
have  different  requirements,  and  their  requirements  may  change  as  they  move  from 
one  level  of  expertise  to  another.  (Goodwin,  1993:230) 

Ease  of  use  is  supported  as  an  integral  element  of  user  acceptance  as  Davis  (1989)  claims 

all  else  being  equal  an  application  perceived  to  be  easier  to  use  than  another  is  more  likely 

to  be  accepted  by  users. 

Functionality 

The  constmct  of  functionality  as  discussed  by  Goodwin  (1993)  refers  to  the 
program  containing  the  functions  needed  to  do  their  tasks.  Functionality  is  equated  to  the 
content  construct  in  the  EUCS  measures  of  DoU  and  Torkzadeh  (1988),  Etezadi-Amoli 
and  Farhoomand  (1996),  and  others.  Etezadi-Amoli  and  Farhoomand  (1996)  found  the 
functionality  to  be  one  of  the  two  most  important  components  of  user  satisfaction.  Davis 


16 


et  al  (1989)  produced  data  to  suggest  that  ease  of  use  may  not  be  as  important  to 
determining  user  acceptance  as  the  construct  of  functionality.  They  hypothesize  the  user 
may  be  willing  to  tolerate  a  difficult  interface  in  order  to  access  functionahty  that  is 
important,  while  no  amount  of  ease  of  use  will  be  able  to  compensate  for  a  system  that 
doesn’t  do  a  useful  task.  Getting  users  involved  in  determining  the  requirements  for 
functionality  and  ease  of  use  is  hypothesized  to  foster  system  acceptance  by  the  users. 

User  involvement  is  discussed  in  the  next  section. 

User  Involvement 

Ives  and  Olson  (1984)  state  user  involvement  refers  to  participation  in  the  system 
development  process  by  representatives  of  the  target  user  group.  Amoako-Gyampah  and 
White  (1993)  found  a  strong  relationship  existed  between  user  involvement  in  system 
development  and  implementation  and  user  satisfaction  with  the  system.  Dodd  and  Carr 
(1994)  propose  the  idea  that  user  involvement  falls  on  a  continuum  ranging  from  little  user 
involvement  as  with  SDLC-based  development  to  the  end  user  developing  the  system  on 
their  own  with  little  or  no  help  from  the  data  services  department.  While  Ives  and  Olson 
(1984),  Amoako-Gyampah  and  White  (1993),  and  Dodd  and  Carr  (1994)  support  the  idea 
that  user  involvement  leads  to  user  acceptance,  the  idea  has  been  debated  heavily  in  the 
research. 

Ives  and  Olson  (1984)  reported  on  six  studies  of  the  user  involvement  related  to 
user  acceptance  provided  little  support  for  user  involvement.  They  concluded  the  benefits 
of  user  involvement  have  not  been  clearly  demonstrated  and  more  research  is  needed. 
Subsequently,  Tait  and  Vessey  (1988)  reported  that  user  involvement  reduces  the  risk  of 
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failure  in  complex  projects.  Amoako-Gyampah  and  White’s  (1993)  results  show  user 
perception  of  level  of  involvement  has  a  direct  positive  and  significant  impact  on  user 
satisfaction. 

Summary 

The  systems  development  life  cycle  provides  an  organized  approach  to  create  new 
systems.  There  many  different  interpretations  of  the  SDLC  to  organize  development. 
Martin’s  twelve  step  model  provides  a  useful  breakout  of  the  steps.  The  current  research 
will  determine  the  requirements  for  a  proposed  on-line  publishing  system  for  the  Air 
Force.  Requirements  analysis  is  a  difficult  process  of  communicating  the  users  needs  to 
the  system  developers.  The  literature  supports  the  concept  that  meeting  the  users  needs 
for  functionality  and  ease  of  use  will  help  foster  their  acceptance  of  the  system.  Getting 
the  users  involved  in  determining  the  system  requirements  is  argued  to  facilitate  user 
acceptance. 
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III.  Methodology 


Introduction 

It  is  generally  accepted  that  identifying  the  user  requirements  is  one  of  the  first 
steps  in  developing  successful  information  systems  (Byrd  et  al,  1992).  Clarifying  and 
defining  client  expectations  is  one  of  the  most  difficult  and  important  parts  of  the 
requirements  definition  process  (Jeffrey  and  Putman,  1994).  This  study  proposes  getting 
the  users  involved  in  the  early  stages  of  developing  the  on-line  publishing  system  for  the 
management  of  the  United  States  Air  Force  publications. 

A  common  approach  to  defining  the  users’  expectations  is  to  gather  information 
about  the  client  organization,  develop  a  list  of  functions  needed  or  desired,  and  then 
prioritize  this  list  (Jeffrey  and  Putman,  1994).  The  exploratory  nature  of  the  research 
supports  the  use  of  qualitative  research  methods.  This  study  will  use  Miles  and 
Huberman’s  (1984)  model  for  qualitative  research  as  discussed  in  the  following  section. 
This  study  will  utilize  focus  groups  of  potential  users  to  determine  the  requirements  and 
prioritize  them.  Focus  groups  are  discussed  later  in  this  chapter. 

This  chapter  will  introduce  a  model  suggested  by  Miles  and  Huberman  (1984)  for 
performing  qualitative  research,  and  show  the  process  the  researcher  used  to  perform  the 
study  in  accordance  with  the  methodology.  The  methodology  included  four  components: 
data  gathering,  data  analysis,  data  display,  and  conclusion  drawing/verifying.  The 
researcher  used  focus  groups  for  the  data  gathering  component  of  the  model.  A  pilot 
smdy  was  performed  to  ensure  the  instrument  was  performed  to  ensure  the  data  gathering 
instrument  was  adequate.  The  data  analysis  was  performed  using  the  model  introduced  in 
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the  bin  creation  section  of  this  chapter.  Chapter  IV  will  display  the  data  gathered  by  the 
focus  groups.  Chapter  V  will  be  used  to  document  the  conclusions  drawn  and  verified  by 
the  researcher  about  the  data. 

Data  Analysis  Methodology 

The  analysis  of  the  data  is  an  essential  element  of  the  research  project,  and  will  be 
performed  using  the  model  represented  in  Figure  2  as  proposed  by  Miles  and  Huberman 
(1984)  for  analyzing  qualitative  data.  The  model  shows  qualitative  research  as  a  four  step 
model  including:  data  collection,  data  reduction,  data  display,  and  conclusion 
drawing/verifying.  The  authors  intended  the  model  to  represent  interaction  between  the 
four  nodes  to  show  the  process  is  not  linear.  Data  reduction,  data  display,  and  conclusion 
drawing/verifying  are  the  steps  associated  with  data  analysis,  and  can  be  performed  at  any 
time  during  the  research  process. 

Data  Gathering.  Prior  to  the  data  gathering  the  researcher  following  Miles  and 
Huberman’ s  methodology  developed  a  conceptual  framework  for  the  constructs  of 
concern  to  the  present  research.  The  conceptual  framework  is  represented  pictorially  in 
Figure  3  and  is  discussed  later  in  this  chapter.  In  addition  to  the  conceptual  framework, 
the  researcher  decided  upon  some  investigative  questions  prior  to  the  data  gathering  phase 
of  the  research.  Miles  and  Huberman  (1984)  support  these  actions.  They  argue  the 
researcher  should  bring  to  the  field  some  orienting  ideas,  foci,  and  tools.  This  should  not 
limit  the  researcher  with  preexisting  concepts  of  what  they  will  find.  The  data  gathering 
for  this  research  will  be  described  later  in  this  chapter  under  the  Procedure  section. 
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Components  of  Data  Analysis:  Interactive  Model 


Figure  2.  Interactive  Model  of  Data  Analysis 

Data  Reduction.  Data  reduction  refers  to  the  process  of  selecting,  focusing, 
simplifying,  abstracting,  and  transforming  the  “raw”  data  that  appear  in  written-up  field 
notes  (Miles  and  Huberman,  1984).  Building  on  the  conceptual  framework  the  researcher 
has  chosen,  the  data  can  be  categorized  in  groups  or  “bins”  as  Miles  and  Huberman  (1984) 
refer  to  them.  Grouping  the  ideas  generated  by  the  users  will  aid  in  the  process  of 
analyzing  the  data.  There  will  be  bins  for  the  ease  of  use  and  functionality  constructs 
included  in  the  center  of  the  user  acceptance  conceptual  model  from  Figure  3.  The  data 
reduction  for  this  study  will  be  explained  in  the  section  entitled  Bin  Creation. 

Data  Display.  The  ideas  generated  by  the  focus  groups  will  be  listed  under  the 
appropriate  bin  categories  shown  in  Figures  4  and  5.  The  results  will  be  listed  separately 
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for  each  investigative  question  and  for  each  focus  group.  The  data  from  the  individual 
groups  will  be  combined  in  consolidated  bins  for  each  task.  The  data  contained  in  the  bins 
will  be  displayed  in  Chapter  IV. 

Conclusion  DrawingA^erifving.  Conclusion  drawing  starts  at  the  very  beginning  as 
the  researcher  comes  up  with  ideas  about  what  things  mean  at  the  earliest  stages  of  the 
project  (Miles  and  Huberman,  1984).  The  researcher  looks  for  patterns,  regularities, 
explanations,  possible  configurations,  causal  flows,  and  propositions  to  help  reach 
conclusions,  concerning  the  data  gathered  in  the  project.  In  addition  to  drawing 
conclusions,  the  researcher  attempts  to  verify  the  results  through  arguments  supported  by 
the  literature,  discussions  with  other  researchers,  or  with  careful  review  of  the  data  (Miles 
and  Huberman,  1984).  Conclusion  drawing  and  verifying  are  discussed  in  Chapter  V. 

User  Acceptance  Model 

The  conceptual  framework  for  this  study  is  shown  in  Figure  3.  The  model  shows 
user  acceptance  relies  on  the  two  key  constmcts  of  functionality  and  ease  of  use  of  a  new 
information  system.  Previous  research  supports  this  approach  (Davis,  1989;  Goodwin, 
1993;  Doll  and  Torkzadeh,  1988;  and  Etezadi-Amoli  and  Farhoomand,  1996). 
Additionally,  the  earlier  research  supports  the  concept  that  user  involvement  in  the  early 
stages  of  the  project  design  is  of  paramount  importance  in  determiiting  the  requirements 
the  user  has  for  the  functionality  and  ease  of  use.  User  acceptance  is  improved  by  user 
involvement  in  the  determination  of  requirements  for  functionality  of  the  system  and  its 
ease  of  use. 
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Focus  Groups 


According  to  Krueger  (1988),  a  focus  group  can  be  defined  as  a  carefully  planned 
discussion  designed  to  obtain  perceptions  on  a  defined  area  of  interest  in  a  permissive, 
nonthreatening  environment.  He  lists  five  characteristics  or  features  of  a  focus  group:  (a) 
people,  who  (b)  possess  certain  characteristics,  (c)  provide  data  (d)  of  a  qualitative  nature 
(e)  in  a  focused  discussion  (Krueger,  1988).  Morgan  (1988)  argues  that  the  results  of 
focus  groups  can  stand  on  their  own  as  valid  research. 


The  data  from  the  focus  groups  will  be  used  to  answer  the  investigative  questions  raised  in 
Chapter  I. 

1.  What  functions  should  the  system  offer  the  user  in  terms  of  the  work 
environment  and/or  the  desktop  environment? 
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2.  On  which  types  of  items  should  the  user  be  able  to  query  the  publication 
system:  i.e.  key  terms,  subject  areas  or  other  ideas? 

3.  What  should  the  graphical  interface  for  the  on-line  publications  system  look 
like? 

The  next  section  discusses  the  characteristics  of  the  people  who  participated  in  the  focus 
groups. 

Participants 

The  participants  in  the  focus  groups  were  volunteers  from  the  students  and  staff  of 
the  Air  Force  Institute  of  Technology  (AFTT),  who  come  from  all  areas  of  the 
organization.  The  volunteers  were  solicited  through  a  briefing  to  the  AFTT  administrative 
staff  at  the  Commandant’s  weekly  staff  meeting.  The  researcher  asked  for  students  and 
staff  members  who  were  interested  in  providing  input  from  the  user’s  perspective  about 
the  requirements  for  an  on-line  publications  system  for  the  United  States  Air  Force. 
Volunteers  were  sought  who  were  experienced  with  various  computer  interfaces  and/or 
the  Air  Force  publications  system. 

The  groups  in  this  study  met  Krueger’s  characteristics  that  define  the  concept  of  a 
focus  group.  They  were  made  up  of  (a)  people,  (b)  who  would  be  users  of  the  proposed 
system,  (c)  they  would  provide  data  (d)  of  a  qualitative  nature,  and  (e)  be  led  through  a 
focused  discussion.  The  determination  of  the  size  of  each  group  is  discussed  in  the  next 
section. 

Group  Size.  The  size  of  the  group  can  affect  its  productivity  (Tubbs,  1992).  Too 
few  members  limits  the  number  of  ideas  the  group  can  come  up  with,  and  even  numbers  of 
individuals  makes  it  hard  to  make  decisions  as  tie  votes  are  possible.  Larger  groups  tend 
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to  spend  too  much  time  organizing  and  get  distracted  from  the  original  goal.  Bales  (1954) 
suggests  five  is  the  optimum  size  for  small  groups  to  function  most  efficiently.  Krueger 
(1988)  suggests  focus  groups  are  typically  seven  to  ten  people,  but  range  from  four  to 
twelve.  The  researcher’s  lack  of  experience  as  a  facilitator  and  the  arguments  by  Bales 
and  Tubbs  determined  five  would  be  the  most  productive  and  manageable  size  for  the 
group.  Five  is  within  the  acceptable  range  for  focus  groups  of  4  to  12. 

Pilot  Study.  The  participants  demographics  are  shown  in  Table  1.  The  group  was 
composed  of  three  males  and  two  females.  The  rank  stmcture  of  the  group  was  four 
Captains  and  one  Major.  The  scale  for  the  remaining  statements  on  the  Personal 
Information  Worksheet  asked  the  respondent  to  report  a  numerical  score  from  one  for 
disagree  to  seven  for  agree.  The  members  reported  an  average  of  5.8  for  the  statement,  “I 
consider  myself  experienced  with  the  general  use  of  computers.”  They  reported  an 
average  score  of  5  for  the  statement,  “I’ve  experienced  many  different  types  of  computer 
interfaces.”  On  the  final  statement,  “Fm  an  experienced  user  of  USAF  publications,”  the 
group’s  average  score  was  3.8.  Two  members  reported  scores  of  one,  while  the  other 
group  members  had  significantly  higher  scores  of  five,  five,  and  seven. 


Table  1.  Demographic  Data  for  Pilot  Study 


Member 

Gender 

Rank 

Computer 

Experience 

Interface 

Experience. 

Phiblications 

Experience 

1 

Male 

Capt 

6 

6 

5 

2 

Female 

Capt 

5 

4 

1 

3 

Male 

Maj 

4 

3 

5 

4 

Female 

7 

6 

7 

5 

Male 

_ Capt _ 

7 

6 

1 

Average 

5.8 

5 

3.8 
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Group  One.  The  first  focus  group  was  comprised  of  three  candidates  for  master’s 
degrees  from  the  December  1997  AFIT  class  and  two  librarians  from  the  AFIT  Library. 
The  demographics  for  this  group  appear  in  Table  2.  The  group  was  comprised  of  three 
females  and  two  males,  three  Air  Force  officers  and  two  Civil  Servants.  The  data  from  the 
worksheet  shows  the  members  considered  themselves  experienced  in  the  general  use  of 
computers  as  the  average  score  for  the  group  was  6.0  on  the  same  scale  as  discussed  in 
the  pilot  study  group.  They  were  not  as  experienced  with  different  interfaces  and  with  AF 
publications  as  the  average  score  on  these  two  items  was  4.4. 


Table  2.  Demographic  Data  for  Group  One 


Member 

Gender 

Rank 

Computer 

Experience 

Interface 

Experience. 

Publications 

Experience 

1 

Male 

ILt 

7 

6 

5 

2 

Male 

Capt 

7 

4 

5 

3 

Female 

GS-10 

6 

6 

3 

4 

Female 

GS-9 

4 

4 

5 

5 

Female 

Capt 

6 

2 

4 

Average 

6 

4.4 

4.4 

Group  Two.  The  second  focus  group  was  comprised  of  three  members  of  the 
Communications  Directorate  at  AFIT,  a  secretary  in  the  Logistics  and  Acquisition  School 
of  AFIT,  and  a  master’s  candidate  in  the  December  1996  AFIT  IRM  program.  The 
demographics  for  this  group  are  displayed  in  Table  3.  The  group  reported  an  average 
score  of  6.2  for  the  statement,  “I  consider  myself  experienced  in  the  general  use  of 
computers.”  The  group  average  of  4.6  showed  they  were  less  confident  they  had 
experience  with  different  computer  interfaces.  The  final  question  rated  the  members’ 
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reported  experience  with  the  AF  publishing  system.  The  team  reported  an  average  score 
of  4.8. 


Table  3.  Demographic  Data  for  Group  Two 


Member 

Gender 

Rank 

Computer 

Experience 

Interface 

Experience. 

Publications 

Experience 

1 

Male 

4 

2 

6 

2 

Female 

GS-5 

6 

4 

3 

3 

Male 

7 

7 

5 

4 

Male 

2Lt 

7 

6 

3 

5 

Female 

SRA 

7 

4 

7 

Average 

6.2 

4.6 

4.8 

Procedure 


Upon  arrival,  the  participants  were  greeted  and  seated  at  desks  positioned  in  a 
semi-circle  with  the  open  side  facing  a  flip-pad  and  a  dry-erase  board.  They  were  asked  if 
they  objected  to  the  project  being  video  taped  for  future  analysis  of  the  session.  There 
were  no  objections,  but  the  research  would  have  used  an  audio  recording  as  a  substitute. 
They  were  read  a  scripted  brief  of  the  background  behind  the  project  that  is  included  at  the 
top  of  Appendix  A.  The  general  procedure  was  scheduled  for  116  hours.  The  various 
tasks  are  listed  below  followed  by  a  discussion  of  the  detailed  procedures  for  each  focus 
group. 

1 .  The  participants  were  given  a  handout  (see  Appendix  A)  that  contained  a 
paragraph  concerning  the  project  scope  and  an  agenda  with  approximate  times 
for  completing  each  task.  They  were  asked  to  read  the  project  scope  and  ask 
any  questions  they  had  at  ±at  point  about  the  project. 

2.  The  first  task  the  participants  were  required  to  complete  was  to  fill  out  a 
personal  information  sheet  (see  Appendix  B).  This  task  was  scheduled  for  3 
minutes  and  asked  for  some  basic  information  about  them  and  their  experience 
with  computers  and  AF  Publications. 
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3.  Following  the  personal  information  sheet,  the  participants  were  given  five 
minutes  to  work  as  a  team  to  work  on  task  two;  a  copy  of  the  task  is  included 
as  Appendix  C  (Scannell  and  Newstrom,  1991).  The  task  required  the 
members  to  think  creatively  to  come  up  with  the  answers  to  the  puzzle.  The 
task  was  intended  to  be  an  ice  breaker,  foster  group  cohesiveness,  and  act  as 
an  aid  to  creativity. 

4.  For  task  three,  the  participants  brainstormed  for  30  minutes  on  investigative 
question  one,  “what  functions  should  the  system  offer  the  user  in  terms  of  the 
work  environment  and/or  the  desktop  environment?”  They  were  encouraged 
to  ask  for  clarification  if  they  didn’t  understand  what  they  were  being  asked  to 
do.  The  researcher  acted  as  the  recorder,  writing  the  ideas  generated  by  the 
group  on  the  flip-chart.  As  the  pages  were  filled  with  ideas  they  were  removed 
from  the  flip-chart  stand  and  put  up  on  the  wall  for  the  participants  to  see  as 
they  continued  brainstoraiing. 

5.  The  participants  were  then  given  10  minutes  for  task  four  to  brainstorm  ideas 
about  investigative  question  two,  “on  which  types  of  items  should  the  user  be 
able  to  query  the  publication  system:  i.e.  key  terms,  subject  areas  or  other 
ideas?”  Again  the  author  acted  as  the  recorder  and  the  same  procedure  was 
used. 

6.  The  participants’  fifth  task  was  to  use  the  dry-erase  board  and  the  flip-chart  to 
come  up  with  a  depiction  of  what  the  screen  should  look  like  to  answer 
investigative  question  three,  “what  should  the  graphical  interface  for  the  on¬ 
line  publications  system  look  like?”  They  were  encouraged  to  select  someone 
to  take  the  markers  and  begin  to  draw  some  mock  screens  of  what  they  wanted 
the  system  to  look  like.  They  had  30  minutes  for  this  task. 

7.  The  final  task  requested  of  the  group  was  to  prioritize  the  ideas  generated  by 
task  three  and  four.  They  were  each  given  50  Post-it  ®  notes  to  vote  for  the 
functions  they  felt  were  the  most  important  and  50  to  vote  for  the  most 
important  types  of  things  for  which  the  user  should  be  able  to  query  the 
system.  They  were  given  10  minutes  to  place  the  sticky  notes  directly  on  the 
flip  chart  pages  with  the  listed  ideas  to  cast  their  votes. 

Pilot  Group 

Focus  Group  Introduction.  The  first  focus  group  was  performed  as  a  pilot  study 
using  a  group  composed  of  people  from  the  masters  program  for  information  resource 
management  (IRM)  graduating  from  the  Air  Force  Institute  of  Technology  (AFIT)  in 
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December  1996.  The  group  was  homogenous  as  they  were  all  Air  Force  officers 
attending  the  same  school.  They  were  comfortable  together  in  the  very  early  stages  of  the 
group  interaction  as  they  sat  down  and  talked  among  themselves.  The  group  was  quick  to 
ask  questions  about  the  project  and  for  clarification  of  the  instructions.  Overall,  the  group 
was  very  relaxed  and  involved  in  the  process  from  the  beginning 

Task  One:  Personal  Information  Worksheet.  The  first  organized  activity  of  the 
focus  group  was  to  fill  out  the  personal  information  worksheet;  a  copy  is  included  as 
Appendix  B.  One  group  member  had  little  experience  with  publications  and  was  unsure 
what  number  to  put  for  the  answer.  After  a  short  discussion,  the  respondent  chose  an 
answer.  The  group  completed  the  rest  of  the  worksheet.  This  section  was  completed  in 
the  time  allotted. 

Task  Two:  Ice  Breaker.  The  second  task  for  the  group  was  to  attempt  the  word 
puzzle  (see  Appendix  C)  used  in  this  process  as  an  ice  breaker,  team  builder,  and  an  aid  to 
develop  the  creative  energy  of  the  group.  This  group  already  knew  each  other  and  had 
worked  together  on  other  occasions,  so  the  ice  breaker  function  of  the  puzzle  was  not  as 
important  in  this  case  as  team  building  and  fostering  the  groups  creativity.  One  group 
member  did  not  participate  in  the  task,  because  he  didn’t  like  word  puzzles.  However,  the 
member  participated  in  the  rest  of  the  tasks. 

Task  Three:  Functionality.  This  task  required  the  group  to  brainstorm  ideas  for 
investigative  question  one,  “what  functions  should  the  system  offer  the  user  in  terms  of  the 
work  environment  and/or  the  desktop  environment?”  One  of  the  members  asked  for 
clarification  about  what  the  system  was  assumed  to  mean.  The  researcher  explained  the 
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intent  was  to  come  up  with  user  requirements  for  an  on-line  computer  system  to  access 
organizational  information  including  operating  instructions,  directives,  visual  aids,  and 
other  manuals  and  pamphlets. 

The  group  discussed  accessibility,  reliability  and  a  possible  backup  for  the 
proposed  system.  The  group  expressed  concern  about  accessibility  and  downtime.  The 
CD/ROM  system  currently  used  to  access  some  pubhcations  was  used  as  an  example  of 
this  concern.  The  system  required  a  CD/ROM  reader  on  the  computer  and  the  correct 
software  loaded  onto  the  hard  drive.  Several  group  members  talked  about  the  limited 
number  of  computers  available  to  read  the  publications.  The  group  required  the  proposed 
system  to  be  available  on  most  or  aU  computers  in  the  organization. 

The  concern  for  the  new  on-line  system  also  included  the  reliabihty.  The  group 
felt  there  would  be  times  they  would  not  be  able  to  connect  to  the  system  or  it  would  be 
down.  They  wanted  the  system  to  be  more  reliable  than  most  on-line  systems  they  were 
familiar  with.  Also,  any  backup  system  must  have  an  acceptable  performance  level. 
Backup  was  discussed  in  the  form  of  paper  publications  or  the  current  CD/ROM  system. 

'  The  group  discussed  ensuring  the  user  had  the  most  current  version  of  the 
publication.  The  group  members  were  famihar  with  some  current  Internet  sites  that  don’t 
guarantee  the  user  the  most  current  version.  They  assumed  this  should  be  controlled  by 
providing  the  official  site  or  sites  for  the  Air  Force.  In  addition  to  the  latest  version,  the 
users  wanted  to  be  able  to  identify  the  items  that  changed  from  one  version  to  the  next. 
They  required  a  section  for  each  publication  to  be  named  “what’s  new,”  or  suggested  a 
different  color  text  for  the  changes  from  the  last  version. 
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Within  many  publications  are  specific  references  to  other  publications  for  further 
direction  or  clarification.  The  group  required  the  on-line  system  to  move  from  the  point 
of  the  reference  to  the  section  of  the  text  referenced  with  a  click  of  the  mouse.  They 
named  the  idea  hypertext,  and  as  the  discussion  continued  the  group  added  the  ideas  of 
linking  to  relevant  tables,  charts,  pictures,  video  clips,  etc.  The  researcher  suggested  the 
term  hypermedia  for  this  requirement. 

The  group  expanded  the  linking  idea  with  a  term  they  labeled  embedded  forms. 
The  example  given  for  this  idea  was  the  instruction  covering  leave.  They  required  the 
ability  to  pull  up  the  leave  form,  while  reading  the  leave  instruction.  Additionally,  they 
discussed  the  idea  of  a  link  from  each  block  of  the  leave  form  to  the  section  of  text  in  the 
instruction  giving  the  user  specific  directions  on  how  to  fiU  in  the  block.  Furthering  this 
idea,  the  group  decided  a  section  called  frequently  asked  questions,  or  an  e-mail  link  to  the 
office  of  primary  responsibility  (OPR)  for  the  instruction,  could  help  clear  up  any 
questions  not  clearly  answered  by  the  text  of  the  instruction. 

Task  Four:  Query  CapabiUtv.  The  topic  of  a  search  capability  was  brought  up  in 
the  discussion  of  the  previous  question  concerning  the  functionality  of  the  system.  The 
researcher  suggested  further  discussion  of  the  function  should  wait  for  this  section  of  the 
focus  group.  The  discussion  generally  covered  the  ability  for  any  user  to  find  the  desired 
information  regardless  of  their  knowledge  of  search  tactics. 

The  group  members  required  flexibility  in  setting  the  delimiters  for  any  search  they 
would  perform.  They  wanted  to  be  able  to  search  all  Department  of  Defense  publications 
or  narrow  it  down  to  Air  Force,  MAJCOM,  base,  or  other  relevant  levels.  The  group 
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discussed  the  availability  of  other  publications  not  in  the  user’s  current  chain  of  command. 
A  group  member  used  the  example  of  a  person  going  on  a  temporary  duty  from  one 
command  to  another  might  want  to  check  the  uniform  requirements  of  the  command  and 
base  to  which  they  were  going. 

The  group  also  required  the  ability  to  search  publications  with  different  types  of 
queries.  They  expressed  a  requirement  to  perform  searches  for  any  word  related  to  the 
topic  they  wanted  and  get  the  number  of  times  the  word  appears  in  each  publication.  This 
requirement  was  labeled  as  a  hit  count.  Another  option  requested  was  the  ability  to 
perform  a  search  on  designated  keywords.  They  also  wanted  to  be  able  to  query  the 
publications  by  title  or  series  number.  Within  the  publication,  the  group  wanted  the  ability 
to  search  for  a  particular  chapter,  subject,  section,  or  other  levels  to  get  to  the  exact 
information  for  which  they  are  looking.  The  final  type  of  search  the  group  required  was 
labeled  as  a  natural  language  query.  This  entailed  the  ability  of  the  user  to  query  the 
system  for  the  actual  question  they  wanted  answered.  As  an  example  they  mentioned  the 
user  posing  the  question,  “what  do  I  need  to  do  to  go  on  leave?”  They  required  the 
system  give  them  a  list  of  simple  instructions  with  links  to  the  appropriate  forms  and 
instructions  to  get  the  user  ±e  information  to  go  on  leave. 

The  final  idea  under  this  section  was  the  desire  to  maintain  a  history  of  the 
searches  the  user  has  performed  previously.  The  group  felt  the  average  user  would 
perform  the  same  type  of  work  on  more  than  one  occasion,  and  the  same  query  would 
take  them  to  information  they  needed  at  a  later  time.  The  ability  to  save  the  search  results 
or  mark  the  publication  or  section  of  publication. 
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Task  Five:  Screen  Design.  The  researcher  asked  for  a  volunteer  to  draw  the 


group’s  ideas  for  the  way  the  opening  screens  of  the  system  would  look.  The  researcher 
had  to  explain  to  the  group  that  no  artistic  ability  was  necessary  as  only  a  crude  drawing 
was  needed.  With  a  little  delay  and  the  threat  the  researcher  would  have  to  pick  someone, 
a  volunteer  did  step  up  to  the  flip-chart.  Shortly  into  the  initial  drawing,  a  second 
individual  stepped  up  to  continue  the  process. 

The  main  theme  of  this  group  on  the  look  of  the  system  was  the  desire  to  make  it 
look  familiar.  The  requirement  for  this  was  to  make  the  system  look  like  the  Windows, 
Internet  browsers,  or  other  systems  Air  Force  users  are  currently  operating.  They 
determined  this  was  the  way  to  present  the  interface  of  the  system  to  get  the  users  to 
understand  it  with  little  if  any  additional  training. 

Lessons  Learned.  Task  one  worked  well  for  the  intended  purpose,  and  no  changes 
were  made.  Task  two  provided  two  areas  for  improvement  as  the  researcher  determined 
to  act  as  recorder  for  the  puzzle  solutions  and  to  provide  the  group  the  answers  orally. 

The  researcher  wanted  the  groups  to  work  more  on  the  communication  to  solve  the 
puzzles  than  writing  the  answers.  Providing  a  written  copy  of  the  answers  to  each  team 
member,  reduced  the  group  interaction  as  they  read  the  answers  individually.  The 
researcher  had  one  of  the  group  members  record  the  ideas  generated  in  tasks  three  and 
four  for  the  pilot  group.  Having  the  one  group  member  in  front,  changed  their  status  with 
the  rest  of  the  group.  The  researcher  acted  as  recorder  for  the  idea  generation  tasks  for 
the  rest  of  the  groups.  Using  both  the  white  board  and  the  flip  chart  for  recording  the 
drawings  from  task  five  worked  effectively.  Another  change  to  the  procedure  involved 
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adding  the  voting  process  of  task  six.  The  pilot  group  was  not  required  to  vote  on  the 
ideas  generated  in  tasks  three  and  four,  until  after  the  fact.  The  researcher  determined  the 
best  time  to  capture  the  opinion  of  the  group  about  the  ideas  was  at  the  end  of  the  session, 
when  the  thoughts  were  still  fresh  in  their  minds. 

Group  One 

Task  One:  Personal  Information  Worksheet.  The  group  asked  for  no  help  to 
understand  the  worksheet  and  completed  the  task  in  the  allotted  time. 

Task  Two:  Ice  Breaker.  This  task  required  the  group  complete  the  word  puzzle 
(see  Appendix  C)  designed  as  an  ice  breaker,  team  builder,  and  as  an  aid  to  creativity  for 
the  session.  The  researcher  handed  a  copy  of  the  word  puzzles  to  each  of  the  members. 
One  of  the  members  answered  several  of  the  individual  puzzles  on  the  sheet.  The 
researcher  made  several  attempts  to  get  the  others  involved.  The  first  attempt  was  to  ask 
the  members  to  help  decide  if  the  answers  from  the  active  member  were  right.  Next,  the 
researcher  asked  the  panel  to  start  suggesting  some  ideas  for  the  other  puzzles.  The 
attempts  were  successful  as  other  members  began  to  come  up  with  some  ideas  and  suggest 
some  changes  to  other  answers  put  forth. 

Task  Three:  Functionalitv.  The  third  task  required  the  group  to  answer  the  first 
investigative  question;  “what  functions  should  the  system  offer  the  user  in  terms  of  the 
work  environment  and/or  the  desktop  environment?”  This  section  describes  the 
discussion  that  took  place  on  the  topic  of  the  proposed  system’s  desired  functions. 

The  researcher  acted  as  the  recorder  for  this  and  subsequent  idea  generation  tasks. 
The  researcher  was  careful  to  record  only  the  ideas  the  team  generated  and  ask  for 
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guidance  on  the  wording  of  the  phrases  used  on  the  flip-chart  pad.  Additionally,  the 
researcher  informed  the  group  on  several  occasions  to  suggest  corrections  if  the  items 
recorded  did  not  convey  the  meaning  of  the  thoughts  they  were  trying  to  convey. 

At  the  beginning  of  the  discussion,  one  of  the  team  members  asked  for  clarification 
concerning  what  was  included  in  the  term  publications  for  the  discussion.  The  researcher 
defined  the  publications  to  include  directives,  instmctions,  pamphlets,  visual  aids,  and 
other  forms  of  written  information,  but  not  including  technical  orders  as  they  are  not  part 
of  the  purview  of  the  publishing  office  at  this  time. 

One  of  the  members  asked  if  they  could  complain  about  the  CD/ROM  system 
implemented  for  accessing  AF  publications.  The  researcher  answered  the  question  with 
the  statement  to  the  effect  that  they  could  use  what  the  member  felt  was  wrong  to  come 
up  with  suggestions  for  making  the  on-line  system  better  than  the  CD/ROM.  The  member 
then  stated  the  system  needed  to  be  widely  accessible  to  the  AF  users  in  the  field.  This  led 
to  the  requirement  for  the  system  to  be  available  on  virtually  every  computer  being  used 
for  AF  business. 

The  next  topic  of  discussion  generated  the  requirement  to  have  publications 
updated  centrally.  The  group  required  changes  to  publications  be  performed  by  the  OPR, 
and  be  available  to  the  users  with  no  effort  on  their  part.  They  pointed  out  changes  on  the 
system  should  be  implemented  with  no  significant  delay. 

The  group  discussed  the  current  Internet  sites  providing  access  to  some  of  the  AF 
publications.  They  mentioned  the  sites  usually  have  a  disclaimer  warning  the  publications 
found  at  the  site  may  not  be  the  most  current  version  of  the  document.  The  members 
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required  the  on-line  system  assure  the  users  the  publications  contained  on  it  are  the  most 
current  versions  and  contain  all  changes  made  to  date.  They  determined  a  single  official 
AF  publications  system  should  provide  this  security. 

Team  members  expressed  concern  about  an  on-hne  system  being  reliable  enough 
to  count  on  during  day-to-day  operations.  They  didn’t  want  a  computer  or 
communications  problem  to  stop  the  work  process.  They  felt  the  systems  they  were 
familiar  with  endured  periods  of  downtime  or  slow  response.  They  required  a  system  of 
backup  using  paper  or  CD/ROM  be  required  to  keep  things  running  smoothly  during 
system  failures  or  for  deployments. 

Deployments  were  another  area  the  group  was  concerned  with  in  terms  of  an  on¬ 
line  system  being  accessible  and  reliable  enough  for  the  day-to-day  operations  of  the  unit. 
The  researcher  informed  the  group  members  the  Air  Force  Electronic  Publishing  Master 
Program  Guide  called  for  requiring  CD/ROM  as  a  backup  for  the  on-line  system  for  the 
foreseeable  future  (Air  Force,  1996). 

The  experience  some  of  the  team  members  had  with  the  implementation  of  the 
CD/ROM  into  the  AF  units  caused  some  concern  about  how  this  proposed  system  would 
be  implemented.  The  group  required  an  implementation  plan  be  designed  up  front  to 
make  the  switch  to  the  on-line  system  smooth  and  feasible. 

Annual  and  special  updates  of  instmctions  were  the  next  topics  of  discussion.  The 
group  required  the  process  be  accomplished  completely  on-line.  Related  to  this  idea  of 
electronic  routing  of  the  review  process  is  the  electronic  routing  and  approval  of  forms. 
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The  members  determined  the  biggest  hurdle  to  moving  ahead  with  the  electronic  routing 
process  was  the  verification  of  the  approving  official’s  electronic  signature. 

Following  the  discussion  of  electronic  routing,  the  topic  of  hypertext  or 
hyperlinking  of  various  types  of  information  was  discussed.  For  example,  the  group 
required  the  ability  to  link  references  in  the  publication  to  pictures,  graphs,  charts,  or  other 
supporting  material.  Another  example  they  mentioned  was  the  ability  to  hnk  from  the 
instmction  to  a  needed  form  and  vice  versa  for  an  activity  such  as  taking  leave  or  filing  a 
travel  voucher.  This  requirement  was  labeled  as  embedded  forms. 

Similar  to  the  linking  idea  of  the  previous  paragraph,  the  group  wanted  the  system 
to  keep  track  of  the  references  made  in  the  publications.  They  commented  how  many 
discontinued  publications  have  been  referenced  by  other  documents.  The  group  wanted 
the  system  to  track  these  references,  and  notify  OPRs  for  the  publications  in  order  to 
correct  the  soon  to  be  obsolete  references.  This  led  the  group  to  the  idea  of  notifying  the 
OPR  or  the  system  administrator  in  the  case  of  any  errors  or  misunderstandings 
concerning  the  publications  or  the  system.  Group  two  discussed  many  of  the  same  topics 
as  the  pilot  group  and  had  many  similar  ideas. 

Task  Four:  Query  Capability.  The  two  librarians  were  only  occasionally  involved 
in  the  functionality  discussion,  but  were  integral  to  the  discussion  answering  investigative 
question  two,  “what  should  the  graphical  interface  for  the  on-line  publications  system  look 
like?”  The  discussion  covered  several  aspects  of  how  the  query  system  should  be 
implemented. 
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The  focus  group  team  members  began  the  discussion  by  explaining  the  requirement 
to  query  the  index  by  publication  number,  series  number,  or  by  title  of  the  publication. 
Additionally,  the  ability  to  search  within  the  document  for  the  exact  location  of  the 
information  was  required. 

The  team  required  the  word  searches  they  performed  to  be  available  in  both  the 
title  and  text  of  the  pubhcations.  They  wanted  a  measure  of  the  number  of  times  the  word 
appeared  in  the  various  publications.  This  requirement  was  labeled  as  a  hit  count.  This 
also  led  to  some  discussion  about  the  system  providing  a  relativity  factor  for  the  different 
results  returned  from  the  search.  This  would  require  an  estimate  by  the  computer  of  the 
likelihood  the  hits  were  in  the  publication  for  which  the  user  was  looking. 

The  group  discussed  users  with  different  levels  of  experience  being  able  to  use  the 
system  effectively.  The  suggestion  first  leading  the  group  to  this  idea  was  to  include 
options  for  the  user  to  initiate  some  search  functions  such  as  Boolean  operators,  use  of 
wild  cards,  and  other  advanced  search  capabilities.  When  they  realized  this  ability  required 
some  previous  experience,  the  recommendation  was  to  provide  a  natural  language  query 
ability  for  these  users. 

Team  members  required  the  ability  to  control  the  scope  of  the  search.  They 
decided  the  user  should  be  able  to  limit  the  search  in  different  ways.  The  term  they  used 
for  the  this  was  to  establish  flexible  search  delimiters  for  narrowing  the  search  field  to  their 
area  of  interest.  An  example  of  this  discussed  was  determining  to  search  AF  level,  a 
particular  MAJCOM,  and/or  the  base  publications  pertaining  to  their  topic.  Another 
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example  of  this  delimiter  was  deciding  what  types  of  publications  to  search,  like 
instructions  only  or  instructions  and  visual  aids. 

The  group  discussion  moved  to  the  idea  of  the  system  keeping  a  history  of 
previous  searches  the  user  had  performed.  The  idea  behind  this  suggestion  was  that  users 
perform  the  same  job  most  of  the  time  and  the  search  would  be  useful  more  than  one  time. 
Additionally,  the  group  wanted  the  ability  to  mark  a  particular  publication  or  sections  of 
publication  the  user  will  go  to  on  numerous  occasions.  Overall,  the  group  wanted  the 
search  to  be  effective  for  different  users,  offer  flexible  methods  to  limit  the  search, 
maintain  a  personal  history  of  prior  searches,  and  offer  an  effective  display  of  the 
information  the  search  came  up  with. 

Task  Five:  Screen  Design.  The  first  comment  for  the  discussion  about  the  user 
screen  was  a  team  member  wanted  the  system  to  look  like  Windows.  When  asked  by  the 
researcher  about  colors  for  the  screen,  the  members  stated  the  default  colors  for  Windows 
would  be  fine  with  them.  The  other  requirement  about  color  the  group  had  was  for  the 
colors  to  be  chosen  with  enough  contrast  for  the  user  with  a  monochrome  screen  to  use 
the  system. 

The  team  members  required  options  for  allowing  the  system  to  run  faster.  They 
wanted  to  be  able  to  download  the  file  in  text  only  mode  with  a  button  on  the  screen  for 
loading  the  graphics  only  if  they  needed  them. 

The  introduction  screen  the  group  wanted  would  provide  buttons  for  the  key 
options  they  would  use.  They  determined  the  most  important  function  on  the  first  screen 
was  the  search  capability  to  help  the  user  find  the  information  they  needed.  They  wanted 
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the  button  for  help  to  lead  the  user  through  the  options  for  limiting  the  search  to  the 
intended  scope. 

The  group  required  the  search  results  to  link  to  the  different  publications.  The 
user  would  click  on  the  result  they  felt  contained  the  information  they  needed  and  it  would 
open  up  that  document  for  them.  They  could  read  the  entire  document  or  perform  another 
search  to  find  the  specific  information  for  which  they’re  looking.  The  screen  the 
publication  opened  up  on  should  offer  the  user  many  options,  according  to  the  group. 

Some  examples  of  the  requirements  include  being  able  to  print  the  document  or  sections  of 
it  or  put  sections  of  the  text  into  another  document.  The  overall  feeling  toward  screen 
design  was  the  need  for  the  system  to  be  familiar  to  the  user  like  the  Windows  interface 
commonly  used  on  AF  operations. 

Task  Six:  Voting.  The  team  members  were  provided  two  stacks  of  Post-it  ® 
notes,  each  containing  50.  The  researcher  had  the  team  members  put  the  number  of  notes 
representing  their  strength  of  opinion  for  the  individual  ideas  generated.  The  intent  of  this 
activity  was  to  measure  the  group  opinion  as  to  which  functions  were  most  important  for 
the  proposed  system. 

Group  Two 

Task  One:  Personal  Information  Worksheet.  The  personal  infomaation  collected 
for  this  group  is  displayed  in  Table  3.  The  worksheet  was  completed  in  the  allotted  time. 

Task  Two:  Ice  Breaker.  The  researcher  prodded  the  group  by  suggesting  they  say 
their  ideas  out  loud  to  help  the  other  members  work  out  some  of  the  puzzles.  Two  of  the 
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members  made  very  few  inputs  into  the  exercise.  The  researcher  made  attempts  similar  to 
those  in  the  pilot  study  to  get  the  other  members  involved  in  the  task. 

Task  Three:  Functionality.  The  researcher  had  the  group  answer  the  first 
investigative  question,  “what  functions  should  the  system  offer  the  user  in  terms  of  the 
work  environment  and/or  the  desktop  environment?”  The  researcher  asked  the  group  if 
there  were  any  questions  about  the  topic.  When  no  questions  were  asked,  the  researcher 
had  the  group  begin  brainstorming  ideas. 

The  first  topic  discussed  by  the  group  was  the  requirement  for  the  system  to 
provide  the  most  current  version  of  the  document  available.  Later  in  the  discussion,  the 
group  addressed  how  the  single  system,  acting  as  the  official  site  for  obtaining 
publications,  should  provide  the  version  control  the  user  desired. 

System  reliability  was  determined  to  be  a  user  requirement.  The  group  talked 
about  the  fact  users  would  start  saving  copies  of  the  publications  to  their  hard  drives  or 
printing  out  copies,  if  they  thought  the  system  might  break  down  or  not  be  available  for 
some  other  reason.  This  led  a  team  member  to  suggest  the  system  allow  the  user  to  save  a 
copy  to  the  hard  drive,  and  the  system  should  identify  if  the  publication  is  changed  or 
updated. 

The  group  discussed  the  need  for  a  robust  search  capability.  They  required  the 
ability  to  find  specific  information  and  go  to  it.  The  search  would  locate  the  exact  location 
of  the  information  the  individual  wanted  and  open  the  document  at  that  point  In  addition 
to  locating  the  specific  information,  a  member  of  the  group  suggested  the  system  provide 
the  ability  to  download  a  part  of  a  document.  If  everything  the  user  needs  to  perform  their 
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job  is  found  in  a  single  paragraph,  page,  or  chapter,  downloading  the  entire  file  is  a  waste 
of  resources. 

The  idea  of  the  time  and  resources  required  to  download  information  led  the  group 
to  another  idea.  The  group  wanted  the  ability  to  view  the  first  part  of  the  file  without 
actually  downloading  the  whole  thing  could  prevent  this  waste.  They  labeled  this 
requirement  as  a  preview  ability. 

One  of  the  members  asked  what  could  be  done  to  allow  the  user  to  see  a  table  or 
chart  referenced  in  a  document.  The  group  quickly  suggested  using  hypertext  links  could 
accomplish  this  goal.  This  led  a  team  member  to  suggest  the  requirement  for  pointers  and 
links  to  supplements  and  changes  to  the  document  with  which  the  user  is  currently 
working.  Talking  about  changes  led  a  member  to  suggest  a  requirement  for  a  master  list 
of  changes,  updates,  deletions,  and  other  information  about  what  is  happening  with  the 
publications. 

After  a  short  pause,  a  suggestion  was  made  to  ensure  the  system  didn’t  require  the 
user  to  save  a  file  to  a  drive  and  open  a  different  viewer  program  to  look  at  the  document. 
It  was  asked  by  a  member  if  there  could  be  a  function  to  help  the  user  understand  the 
meaning  of  the  text  of  a  publication.  The  group  came  up  with  the  requirement  for  a  help 
function,  and  for  a  link  to  the  OPR  of  the  document. 

Accessibility  of  the  system  was  the  next  topic  discussed.  The  group  determined 
the  system  needed  to  be  available  to  everyone  working  on  AF  business.  The  idea  of 
accessibility  led  the  group  to  discuss  system  backup.  The  group  required  another  method 
to  utilize  publications  in  the  event  of  a  temporary  problem  in  accessing  the  system. 
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Adding  to  the  idea,  was  the  requirement  to  make  the  backup  system  work  in  the  same 
manner  as  the  on-line  system. 

The  next  topic  suggested  in  the  discussion  was  the  need  for  the  interface  to  be  user 
friendly.  The  reasoning  behind  this  suggestion  was  to  allow  an  inexperienced  user  get  the 
information  they  desire.  They  required  the  system  to  lead  the  user  through  the  process  of 
finding  the  information  step-by-step.  Someone  quickly  pointed  out  this  option  could 
prove  to  inhibit  the  productivity  of  an  expert  user.  This  led  to  the  requirement  of  a 
multilevel  help  system  catering  to  users  with  different  levels  of  expertise. 

Following  the  user  friendly  discussion,  the  topic  of  response  dme  for  the  system 
was  introduced.  The  group  was  concerned  with  two  aspects  of  response  time:  (1)  how 
long  it  would  take  to  get  into  the  system  and  (2)  how  long  the  files  took  to  get  to  the  user, 
or  download.  One  option  for  keeping  the  download  time  to  a  minimum  the  group 
suggested  was  to  offer  the  user  of  downloading  the  information  in  a  text  only  format,  and 
only  download  certain  pictures  or  graphics  contained  in  the  document  when  needed.  The 
end  of  the  time  allotted  for  the  section  was  reached  and  the  researcher  asked  the  group  if 
there  was  any  last  items  they  wanted  to  include  in  this  section. 

Task  Four:  Query  Capability.  This  task  had  the  group  answer  investigative 
question  two,  “on  which  types  of  items  should  the  user  be  able  to  query  the  publication 
system:  i.e.  key  terms,  subject  areas  or  other  ideas?”  The  group  had  mentioned  the  idea 
of  a  search  capability  during  the  functionality  section,  and  the  researcher  had  suggested 
they  delay  further  discussion  on  the  topic  until  this  section.  The  group  had  no  questions 
on  what  was  expected  of  them  in  this  section. 
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The  first  thing  the  team  required  was  the  ability  to  query  the  system  on  various 
kinds  of  searches  of  the  index.  In  the  event  the  person  had  some  idea  where  to  find  the 
publication,  they  wanted  to  search  the  index  by  series  number  or  publication  number. 

They  also  wanted  the  user  to  be  able  to  query  the  index  for  keywords,  subject,  or  words 
or  phrases  contained  in  the  titles  when  they  didn’t  have  an  idea  what  they  were  looking. 

In  addition  to  searching  the  index,  the  group  required  the  system  provide  an  abstract  of 
each  document  the  users  could  queiy  on  for  a  keyword,  word,  or  phrase  search. 

The  previous  searches  they  suggested  would  take  the  user  to  the  particular 
publication  for  which  the  user  searched.  The  group  required  the  system  also  help  the  user 
find  the  specific  information  they  needed  within  the  document. 

The  researcher  prodded  the  group  to  address  the  way  the  results  of  a  search  would 
look  as  the  discussion  was  stopped  momentarily.  The  group  required  the  ability  to  know 
how  many  times  a  word  from  a  word  search  was  found  in  each  publication.  This 
requirement  was  labeled  as  a  hit  count.  When  the  search  yields  a  list  of  possible 
publications,  the  user  should  be  able  to  quickly  look  or  scroll  through  the  listed 
publications  to  fmd  the  one  they  want.  If  a  search  presents  a  large  list  of  possible 
publications,  the  user  should  get  an  option  to  narrow  the  previous  search  to  get  a  lower 
number  of  more  relevant  publications.  When  searching  the  text  of  the  document,  the 
group  wanted  the  search  to  take  the  user  to  the  place  in  the  document  where  the  hit 
occurred.  The  hit  or  the  relevant  section  should  be  highlighted  to  help  the  user  find  the 
needed  information.  The  search  results  should  display  the  OPR  for  the  publication  or  the 
user  should  be  able  to  query  to  find  out  who  was  the  OPR.  Some  additional  things 
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required  in  the  search  results  might  include:  the  size  of  the  document  in  pages  and/or  file 
size,  an  abstract,  the  location  with  a  link,  and  some  publishing  information  about  the  item. 

A  member  suggested  the  requirement  for  the  search  function  to  be  easy  to  use  for 
those  who  have  little  experience  performing  queries.  The  discussion  turned  to  providing 
some  more  advanced  search  capability  for  the  more  experienced  user.  Another  suggestion 
was  made  to  provide  some  sort  of  tutorial  for  the  more  inexperienced  user  to  take  them 
through  the  steps  of  a  more  advanced  search.  Relating  to  the  novice  user,  a  member 
suggested  some  users  will  need  the  ability  to  ask  a  question  of  the  system  in  English  to  get 
the  answer  to  their  question.  The  requirement  was  labeled  as  a  natural  language  query. 
They  wrapped  up  the  query  section  with  a  discussion  of  putting  limits  on  the 
universe  to  be  searched.  They  explained  the  user  might  know  the  publication  they’re 
looking  for  was  an  instruction,  so  searching  all  publications  would  take  more  time  than 
was  necessary  to  find  what  they  want.  In  addition,  they  may  have  only  wanted  to  know 
the  base  policy  on  a  topic,  so  they  wouldn’t  have  needed  to  search  the  AF  or  Major 
Command  publications.  Placing  limits  was  labeled  as  search  delimiters. 

Task  Five:  Screen  Design.  The  researcher  had  the  group  make  drawings  to 
answer  investigative  question  three,  “what  should  the  graphical  interface  for  the  on-line 
publications  system  look  like?”  The  group  did  not  produce  a  volunteer  to  do  the  drawing 
right  away.  The  researcher  had  to  explain  no  artistic  ability  was  necessary  to  perform  the 
job.  Once  the  volunteer  stepped  up,  the  task  moved  along  as  planned. 

The  group  determined  the  first  screen  should  designate  the  system  as  the  official 
AF  site  for  electronic  publications.  They  also  expected  a  disclaimer  at  the  bottom  of  the 
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screen  letting  everyone  know  the  site  was  for  official  business  only.  Some  of  the  options 
they  required  on  the  first  screen  included  the  ability  to  view  the  publications  as  text 
only  and  the  beginning  of  the  search  function.  They  felt  the  search  function  would  be  a 
button  with  pull-down  menus  for  choosing  search  and  display  options.  The  results  of  the 
search  should  provide  the  ability  to  narrow  the  search,  if  the  results  of  the  search  were  too 
large. 

As  the  member  was  drawing  the  screen,  the  group  was  not  talking  very  much.  The 
researcher  tried  to  generate  discussion  by  asking  the  group  if  they  felt  the  individual 
drawing  the  screen  had  picked  the  right  color  for  the  screen  being  drawn.  The  group 
agreed  the  color  didn’t  matter  as  long  as  the  screen  presented  enough  contrast  to  see 
everything,  and  the  colors  did  not  detract  from  the  system.  This  led  to  the  requirement  to 
standardize  the  font  to  ensure  it  was  readable  and  consistent. 

The  group  ended  the  discussion  with  the  idea  to  provide  the  user  access  to  the 
searches  previously  performed.  The  idea  is  the  user  will  work  with  the  same  type  of 
information  more  than  once  in  the  performance  of  the  job,  and  would  save  time  by 
preventing  them  from  having  to  repeat  the  same  search  over  and  over. 

Task  Six:  Voting.  The  researcher  provided  the  team  members  with  the  Post-it  ® 
notes  for  voting.  The  members  again  were  allotted  50  votes  for  the  ideas  from  task  three 
and  50  votes  for  the  ideas  from  task  four.  Following  the  data  gathering  section,  the 
analysis  of  the  data  will  follow  a  methodology  suggested  by  Miles  and  Humberman  (1984) 
for  analyzing  qualitative  data.  The  approach  used  is  outlined  in  the  next  section. 
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Bin  Creation 


Miles  and  Huberman  (1984)  support  the  creation  of  bins  as  a  method  of  organizing 
the  data  gathered.  The  intent  of  the  bin  creation  process  is  to  determine  categories  the 
data  will  be  entered  into  to  further  understanding.  The  categories  were  determined  prior 
to  the  data  gathering  from  constructs  developed  by  Davis  (1989).  The  data  was  gathered 
according  to  the  procedure  outlined  earlier  in  this  chapter,  then  analyzed  using  the  video 
recordings  for  sorting  into  the  appropriate  bins  and  displayed  in  Chapter  IV.  The  bins 
were  used  to  categorize  the  data  based  on  the  proposed  system  versus  the  use  of  the 
publications  in  the  paper  format. 

The  strong  support  in  the  literature  for  the  importance  of  ease  of  use  in  user 
acceptance  led  the  researcher  to  include  the  construct  in  the  conceptual  model  developed 
for  the  current  research  effort.  The  ease  of  use  bins,  shown  in  Figure  4,  are  based  on  three 
subcategories  for  ease  of  use  developed  by  Davis  (1989).  The  first  ease  of  use  bin  is 
labeled  physical  effort  as  it  refers  to  the  ideas  the  users  came  up  with  to  reduce  the 
physical  exertion  required  to  get  the  system  to  achieve  the  desired  outcome.  The  second 
bin  is  mental  effort  and  refers  to  the  ideas  the  users  brainstormed  to  reduce  the  need  to 
deduce  or  remember  what  commands  are  available  and  how  to  make  them  work.  The  final 
bin  under  the  ease  of  use  construct  is  labeled  easy  to  learn.  This  bin  is  for  data  that  relates 
to  how  the  user  figures  out  a  function  for  the  first  time  and  remembers  how  to  do  it  again 
in  the  future. 

Physical  effort  as  stated  refers  to  the  physical  exertion  required  to  accomplish  a 
task.  An  example  of  this  is  the  difference  between  using  one  click  of  the  mouse  to 
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accomplish  a  function  and  needing  to  physically  scroll  through  several  screens  using  many 
clicks  of  the  mouse  to  find  the  desired  function.  Another  difference  that  represents 
physical  effort  is  between  the  user  having  the  system  located  on  the  desk  or  needing  to  go 
to  a  special  terminal  somewhere  remote  from  the  work  area. 

Mental  effort  refers  to  the  amount  of  effort  needed  to  remember  how  to  initiate  a 
desired  function.  An  example  of  this  is  the  difference  between  having  a  button  to  click 
with  the  picture  of  a  printer  to  know  how  to  print  versus  having  to  remember  the 
command  to  print  is  depress  the  “Control”  and  “P”  keys  at  the  same  time.  Another 
example  for  this  is  the  ability  to  save  the  work  by  clicking  the  mouse  on  a  button  wi±  an 
intuitive  graphic  versus  knowing  the  function  is  under  a  pull-down  window  labeled  “File.” 

Easy  to  learn  refers  to  the  ability  to  determine  how  to  use  a  function  for  the  first 
time.  An  example  of  this  is  the  intuitive  picture  of  the  function  on  the  button  versus  the 
need  to  look  in  the  written  documentation  for  the  system  to  understand  how  to  perform 
the  task.  Another  example  of  easy  to  leam  is  the  difference  between  having  instructions 
for  a  desired  function  in  an  on-line  help  with  step-by-step  instructions  for  how  to  perform 
the  function  versus  the  help  only  explaining  what  the  function  does.  The  categories  within 
ease  of  use  are  mutually  exclusive. 

Functionality,  as  discussed  in  the  literature  review,  represents  the  abilities  the 
program  offers  the  user  to  help  perform  their  job  effectively.  Davis  (1989)  also  identified 
three  subcategories  for  functionality.  Categorizing  the  data  from  the  pilot  study  used  only 
two  of  the  three  subcategories  and  led  to  the  exclusion  of  the  third  category  from  the 
functionality  bins  as  pictured  in  Figure  5,  and  discussed  below.  Effectiveness  is  the  first 
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bin  under  the  functionality  construct  and  refers  to  the  system’s  ability  to  increase  the 
user’s  effecti-veness  on  the  job.  The  second  functionality  bin  is  productivity  and  refers  to 
the  ability  of  the  system  to  increase  the  productivity  of  the  user  on  the  job  or  create  a  time 
savings.  The  final  subcategory  for  functionality  suggested  by  Davis  (1989)  is  importance 
and  refers  to  the  overall  importance  of  the  system  to  the  user’s  job.  Importance  was  not 
used  as  a  bin  as  it  refers  to  the  system  as  a  whole  and  its  importance  to  the  user’s  job,  but 
doesn’t  refer  to  individual  functions  the  system  offers. 

Job  effectiveness  refers  to  the  system’s  ability  to  increase  the  user’s  effectiveness 
on  the  job.  An  illustration  of  this  is  when  the  user  must  look  up  the  age  of  the  client 
manually  in  the  paper  files,  but  with  the  computer  system  the  user  can  now  simply  make  a 
query  and  get  the  information  desired.  The  task  would  have  been  prohibitive  due  to  a 


Ease  of  Use  Bins 

Phvsical  Effort 

Mental  Effort 

Easv  to  Learn 

Figure  4.  Ease  of  Use  Bins 
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Functionahty  Bins 

Job  Effectiveness 

Productivity 

Figure  5.  Functionality  Bins 


time  constraint  without  the  system,  but  is  now  feasible.  Another  example  is  when  the  user 
may  have  had  to  take  information  from  a  customer  on  a  form  and  input  it  into  the  records 
at  a  later  time,  but  the  system  allows  the  user  to  put  the  information  into  the  computer  as 
the  customer  reveals  it. 

Productivity  represents  the  ability  of  the  system  to  help  the  user  perform  the  same 
task  faster  than  would  be  possible  without  the  function.  The  user  could  read  the  whole 
document  concerning  the  question,  but  would  incur  a  time  savings  if  the  system  could 
locate  the  exact  reference  to  the  question  within  the  document.  This  differs  from 
effectiveness  as  the  task  is  made  faster  with  productivity,  but  would  in  fact  be  a  different 
task  or  not  probable  without  the  function. 
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The  representation  of  the  conceptual  framework  in  Figure  3  shows  interaction 
between  the  ease  of  use  bins  and  the  functionality  bins.  This  represents  a  nonexclusive 
relationship,  since  the  same  data  may  appear  in  both  the  ease  of  use  and  functionality  bins. 

The  researcher  categorized  the  data  generated  from  task  three  of  the  pilot  study. 
Figure  6  contains  data  categories  in  the  ease  of  use  bins,  and  Figure  7  contains  the  data 
categorized  in  the  functionality  bins.  The  researcher  had  a  peer  categorize  the  same  data, 
after  reading  the  bin  creation  section  of  this  chapter,  to  test  the  construct  validity  of  the 
bins.  The  peer  categorization  is  presented  in  Figure  8  and  Figure  9. 

Eight  of  the  eleven  ideas  the  researcher  placed  in  the  physical  effort  bin  agreed 
with  eight  of  the  twelve  ideas  placed  in  the  peer’s  physical  effort  bin.  The  researcher  and 
the  peer  placed  six  ideas  in  the  mental  effort  bin  with  four  being  the  same  items.  The 
researcher  and  the  peer  placed  the  same  idea  in  the  easy  to  learn  bin.  Both  effectiveness 
bins  received  seventeen  ideas  with  fifteen  being  the  same.  The  researcher  place  eight  ideas 
in  the  productivity  bin  and  the  peer  placed  nine  with  six  being  the  same.  Upon  discussing 
the  disagreement  on  the  items  placed  in  different  bins,  the  researcher  determined  the  bin 
creation  section  needed  a  statement  defining  the  frame  of  reference.  A  statement  was 
added  concerning  how  the  ideas  would  be  placed  in  the  bins  under  the  assumption  the 
proposed  system  of  publications  management  was  being  compared  to  the  paper 
publications  system.  The  researcher  and  the  peer  then  performed  data  categorization  on 
the  data  gathered  in  task  four  from  the  pilot  study.  The  bins  from  this  categorization  for 
the  researcher  are  presented  as  Figure  10  and  Figure  11.  Figure  12  and  Figure  13  depict 
the  peer’s  categorization  of  the  data  from  task  four  of  the  pilot  study. 
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The  researcher’s  physical  effort  bin  contained  eleven  ideas  and  the  peer’s  had  ten 
with  agreement  on  ten.  The  mental  effort  bins  were  empty  for  both  individuals.  Both  easy 
to  leam  bins  contained  the  same  two  items.  The  job  effectiveness  had  fifteen  items  in 
agreement  with  the  researcher  having  one  extra  item  the  peer  didn’t.  The  peer  included 
one  item  in  the  productivity  bin  the  researcher  didn’t,  and  they  agreed  on  the  one  other 
item  in  the  bin.  The  addition  of  the  frame  of  reference  sentence  in  the  bin  creation  section 
increased  the  agreement  for  the  second  evaluation. 
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Figure  6.  Ease  of  Use  Bins  for  Pilot  Group,  Task  3 
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Figure  7.  Functionality  Bins  for  Pilot  Group,  Task  3 
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Figure  8.  Peer  Ease  of  Use  Bins  Pilot  Group,  Task  3 
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Figure  9.  Peer  Functionality  Bins  for  Pilot  Group,  Task  3 
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Figure  11.  Functionality  Bins  for  Pilot  Group,  Task  4 
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Figure  13.  Peer  Functionality  Bins  for  Pilot  Group,  Task  4 


IV.  Results 


Introduction 

The  ideas  generated  in  tasks  three  and  four  by  the  groups  are  displayed  using  the 
bins  discussed  in  Chapter  III.  The  ideas  were  categorized  according  to  the  procedure 
discussed  in  Chapter  III  under  the  Bin  Creation  section  which  utilized  peer  evaluation  of 
the  categorization  process  to  validate  the  effort.  There  are  ease  of  use  and  functionality 
bins  for  each  group  and  each  task.  The  ideas  are  ordered  within  the  bin  categories  from 
the  highest  vote  total  at  the  top  to  the  lowest  at  the  bottom.  In  addition  to  the  individual 
group  bins  for  each  task,  bins  for  the  combined  groups  for  each  task  are  displayed.  The 
data  was  clustered  into  similar  types  in  the  combined  group  bins.  This  format  adds  order 
to  the  data  collected  and  facilitates  drawing  conclusions  about  the  users’  requirements. 
This  chapter  will  also  include  the  researcheK’  representation  of  the  screens  drawn  by  the 
groups  in  task  five  of  the  procedure,  and  a  brief  discussion  of  the  figures. 

Group  One.  Task  Two 

The  task  was  successful  as  an  ice  breaker,  as  a  team  builder,  and  as  an  aid  to 
increasing  the  group’s  creativity.  The  members  began  talking  and  working  together 
during  the  exercise.  The  puzzles  required  the  creative  thinking  to  figure  out  the  correct 
answers. 

Group  One.  Task  Three 

The  ease  of  use  bins  for  the  ideas  generated  by  group  one  for  task  three  are 
displayed  in  Figure  14.  As  stated  before  the  ideas  are  ordered  within  the  bins  from  the 
highest  to  the  lowest  number  of  votes.  The  reason  for  the  ordering  is  to  identify  the  ideas 
the  group  members  determined  were  the  most  important  requirements  for  system  to  have. 
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All  the  bins  for  the  individual  groups  are  ordered  this  way.  The  functionality  bins  are 
display  in  Figure  15.  Some  of  the  same  data  items  appear  in  the  functionality  bins  as  the 
ease  of  use  as  the  conceptual  model  suggests  this  interaction.  The  next  section  displays 
the  data  from  group  one  performing  task  four. 
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Figure  14.  Ease  of  Use  Bins  for  Group  1,  Task  3 


Group  One.  Task  Four 


This  section  contains  the  data  for  the  brainstorming  session  by  group  one  for  task 
four.  The  ideas  categorized  into  the  ease  of  use  bins  for  the  task  are  displayed  in  Figure 
16.  The  functionality  bins  for  the  same  task  are  displayed  in  Figure  17.  The  data  are 
ordered  as  described  earlier  for  the  single  group  bins. 
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Figure  15.  Functionality  Bins  for  Group  1,  Task  3 


Ease  of  Use  Bins 

Physical  Effort 

Automatic  citation  for 
text  cut  and  pasted  30 

Word  search  23 

Search  by  publication 
name  and  number  23 

Mark/^rint  sections  21 

Hypertext  1 8 

Phrase  search  17 

Wild  card  search  14 

Mental  Effort 

Search  history  22 

Hit  count  12 

Easv  to  Learn 

Natural  language  or 
advance  search  22 

On-line  help  1 4 

Figure  16.  Ease  of  Use  Bins  for  Group  1,  Task  4 
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Figure  17.  Functionality  Bins  for  Group  1,  Task  4 

Group  Two.  Task  Two 


Task  two  for  group  two  had  similar  results  as  it  did  for  group  one.  Although,  it 
was  not  quite  as  successful  with  two  of  the  members  not  participating  much  in  the  task. 
This  didn’t  affect  the  overall  group  discussion  as  all  were  involved  in  the  other  tasks  they 
were  asked  to  perform. 

Group  Two.  Task  Three 

This  section  contains  the  data  for  the  brainstorming  session  by  group  two  for  task 
three.  The  ideas  categorized  into  the  ease  of  use  bins  for  the  task  are  displayed  in  Figure 
18.  The  functionality  bins  for  the  same  task  are  displayed  in  Figure  19.  The  data,  as  in  the 
previous  section,  are  ordered  as  described  earlier  for  the  single  group  bins. 
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Group  Two.  Task  Four 


This  section  contains  the  data  for  the  brainstorming  session  by  group  two  for  task 
four.  The  process  was  the  same  as  the  previous  three  sections  for  the  single  group  bins. 
The  ease  of  use  bins  for  the  ideas  brainstormed  by  group  two  for  task  four  are  displayed  in 
Figure  20.  The  functionality  bins  for  the  task  are  displayed  in  Figure  21 .  The  single  group 
displays  will  help  to  create  the  combined  group  bins  discussed  in  the  next  two  sections. 
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Figure  18.  Ease  of  Use  Bins  for  Group  2,  Task  3 


All  Groups.  Task  Three 


This  section  and  the  next  combines  the  data  from  both  groups  into  a  single  set  of 
ease  of  use  and  functionality  bins  for  task  three  and  four.  In  addition  to  sorting  the  data 
from  highest  to  lowest  vote  totals,  the  ideas  in  the  combined  bins  are  also  sorted  into 
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clusters  of  related  data.  The  ideas  are  sorted  from  highest  to  lowest  vote  totals  first  for 
the  clusters  of  data,  then  the  rest  of  the  ideas  are  listed  from  the  highest  to  lowest  votes 
received.  Figure  22  displays  the  combined  ease  of  use  bins  with  the  data  for  both  groups 
for  task  three. 
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Figure  19.  Functionality  Bins  for  Group  2,  Task  3 


62 


Ease  of  Use  Bins 

Physical  Effort 

Search  text  by 
keyword  65 

Search  index  by  keyword, 
subject,  phrase,  and 
publication  number  60 

Mental  Effort 

Search  delimiters  10 

Search  history  5 

Easy  to  Learn 

Novice  to  expert  search  with 
tutorials  for  novices  35 
Natural  language  search  20 

Figure  20.  Ease  of  Use  Bins  for  Group  2,  Task  4 


Functionality  Bins 

Job  Effectiyeness 

Productivity 

Search  text  for  keywords 

Search  index  by  keyword,  subject. 

65 

Search  history 

5 

publication  number,  etc. 

60 

Search  result 

35 

Query  as  to  OPR 

20 

Search  delimiters 

10 

Hit  count 

0 

Search  goes  to  specific  point 

0 

Figure  21.  Functionality  Bins  for  Group  2,  Task  4 
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The  task  three  data  placed  in  functionality  bins  are  displayed  in  same  format  in  Figure  23. 
Accessibility  and  hyperlink  are  clusters  which  received  votes  from  both  groups  under  the 
physical  effort  category.  No  clusters  were  identified  under  the  mental  effort  category. 

Two  entries  in  the  easy  to  learn  bin  formed  a  cluster  with  six  and  two  votes  for  the  idea. 
The  same  clusters  from  the  physical  effort  bin  are  present  in  the  job  effectiveness  bin  under 
functionality.  The  rest  of  the  ideas  not  in  clusters  follow  in  order  from  most  to  least  votes. 


Ease  of  Use  Bins 

Physical  Effort 

Accessibility 

Widespread  62 

Broad  10 

Hyperlink 

Various  items  20 

General  5 

Control 

Version  12 

Update  5 

Automatic  update  31 

Search  capability  25 

Incorporate 
supplements  15 

Print  ability  8 

Preview  ability  7 

No  additional  programs 
needed  5 

Mental  Effort 

User  friendly  50 

Access  to  OPR  17 

User  preference  history  0 

Easy  to  Learn 

Help 

Context  sensitive  6 

General  2 

Figure  22.  Ease  of  Use  Bins  for  AH  Groups,  Task  3 
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Functionality  Bins 

■Tob  Effectiveness 

Productivity 

Accessibility 

Smooth  and  feasible  implementation 

13 

Widespread 

62 

Preview  ability 

7 

Broad 

10 

Broken  down  into  chapters 

5 

Control  system 

7 

Text  version 

2 

Hyperlink 

Electronic  update 

0 

General  capability 

5 

General  capability 

5 

Reliability 

65 

Automatic  update 

31 

Search  capability 

25 

Deployment  capability 

20 

Access  to  OPR 

17 

Update  process  routed  on-line 

16 

Centralization 

16 

Navigation 

15 

Response  time 

15 

Incorporate  supplements 

15 

POC  for  errors 

,  13 

Date  of  publication/change  directory 

11 

Print  abiUty 

8 

Joint  onerabilitv 

8 

Figure  23.  Functionality  Bins  for  All  Groups,  Task  3 
All  Groups.  Task  Four 


The  clustering  and  ordering  techniques  from  the  previous  section  were  also  used 
for  the  data  gathered  in  task  four  in  this  section.  The  ease  of  use  bins  for  this  task  are 
displayed  in  Figure  24.  There  are  four  clusters  of  data  in  this  set  of  bins.  Text  search  and 
index  search  form  clusters  in  the  physical  effort  bin,  search  history  is  a  cluster  under  the 
mental  effort  bin,  and  a  help  function  forms  the  last  cluster  under  the  easy  to  leam  bin. 
The  rest  of  the  ideas  are  listed  in  order  from  most  to  least  votes  under  the  clusters.  The 
functionality  bins  for  the  same  task  are  displayed  in  Figure  25.  The  job  effectiveness  bin 
under  functionality  contains  four  clusters  including  the  requirement  to  search  the  text, 
search  the  index,  set  search  delimiters,  and  for  the  search  result  to  provide  a  hit  count. 
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Ease  of  Use  Bins 

Physical  Effort 

Menial  Effort 

Easy  to  Learn 

Search  text 

Search 

Help 

Ke3fword 

65 

History  22 

Context  sensitive  6 

Word 

23 

History  5 

General  2 

Search  index 

Hit  count  12 

Keyword,  phrase, 

Search  Delimiters  10 

subject,  and  name  and 

number  of  pub. 

By  publication 

60 

name  and  number 

23 

Automatic  citation 

31 

Mark/print  sections 

21 

Hypertext 

18 

Phrase 

17 

Wildcard 

14 

Figure  24.  Ease  of  Use  Bins  for  All  Groups,  Task  4 


Functionality  Bins 

Job  Effectiveness 

Productiyity 

Search  text 

Search  history 

5 

Text  for  keywords 

65 

Personalized  bookmarks 

4 

Word 

23 

Search  index 

Forpublication  name/number. 

keyword,  subject 

60 

By  publication  name  and  number 

23 

Search 

Delimiters 

32 

Delimiters 

10 

Search  result 

General  result 

35 

Hit  count 

12 

Hit  count 

0 

Natural  language  or  advanced 

22 

Query  at  to  OPR 

20 

Phrase 

17 

Wild  card 

14 

Goes  to  specific  point  in  document 

0 

Figure  25.  Functionality  Bins  for  All  Groups,  Task  4 
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AH  Groups.  Task  Five 


This  section  discusses  the  drawings  and  the  discussion  from  the  groups  concerning 
the  third  investigative  question,  “What  should  the  graphical  interface  for  the  on-line 
publications  system  look  like?”  Group  one  suggested  the  interface  should  look  and  work 
like  the  Windows  the  majority  of  AF  users  are  already  familiar.  The  researcher’s 
representation  of  the  screens  they  drew  are  shown  in  Figure  26.  The  first  screen  group 
one  drew  depicted  the  form  to  be  used  searching  the  system.  The  screen  had  a  space  for 
inputting  the  delimiters  for  the  search  with  a  pull-down  menu  to  aid  the  process.  The 
phrase,  keyword,  subject,  or  other  search  term  had  a  blank  to  be  input.  There  were 
buttons  to  provide  the  common  functions  the  group  expected  the  user  to  need. 

The  other  screen  the  group  drew  was  the  display  of  a  publication  retrieved  by  the 
search.  The  screen  displayed  the  name  and  number  of  the  publication  and  a  large  section 
for  viewing  the  text  of  the  document.  The  top  of  the  page  contained  buttons  for 
performing  the  tasks  the  group  expected  to  perform  often.  Examples  of  the  common 
functions  include:  performing  another  search,  change  the  text  font,  print,  text  only  option, 
and  help. 

Group  two  had  very  similar  screen  formats  as  shown  in  Figure  27.  The  top  of  the 
screen  included  the  designation  as  the  official  AF  publications  site.  The  search  screen 
included  blank  areas  to  input  the  publication  type  to  search,  a  format  for  the  results,  and 
the  term  to  query  upon.  These  input  areas  included  pull-down  menus  for  options  to  help 
the  user  perform  the  search.  The  screen  also  included  some  buttons  for  performing  the 
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common  options  the  user  would  need.  The  second  screen  this  group  drew  was  the  actual 
result  of  the  search.  The  text  area  represented  the  options  that  met  the  criteria  of  the 
search  performed  and  the  bottom  included  an  option  to  perform  the  search  again  with 
narrower  delimiters  to  reduce  to  number  of  hits  from  the  first  search. 

Summary 

The  intention  of  the  data  display,  according  to  Miles  andHuberman  (1984),  is  to 
provide  a  format  for  the  data  that  fosters  the  researcher’s  understanding.  Ordering  and 
clustering  are  two  techniques  they  suggest  for  organizing  the  data.  These  techniques  have 
set  the  stage  for  drawing  and  verifying  conclusions  based  on  the  data.  Chapter  V  will 
detail  the  conclusion  drawing  and  verifying  for  each  investigative  question,  discuss  the 
limitations  of  the  research,  and  make  suggestions  for  related  research  to  be  performed  in 
the  future. 
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V.  Conclusions  and  Recommendations 


Introduction 

The  objective  of  the  research  effort  as  outlined  in  Chapter  I  was  to  answer  the 
question,  what  can  an  on-line,  electronic  publishing  system  offer  users  to  make  the  switch 
from  the  current  system  easier  to  deal  with?  This  Chapter  will  address  each  of  the 
investigative  questions  raised  in  Chapter  I.  In  addition  to  answering  the  questions,  the 
chapter  will  discuss  the  limitations  the  research  experienced.  The  final  section  will  discuss 
two  areas  for  further  research  identified  in  this  study. 

Miles  and  Huberman  (1984)  recommend  twelve  different  methods  for  drawing  and 
verifying  conclusions  including:  counting,  noting  patterns,  seeing  plausibility,  clustering 
like  items,  making  metaphors  to  generalize  particulars,  splitting  variables,  subsuming 
particulars  into  the  general,  factoring,  noting  relations  between  variables,  finding 
intervening  variables,  building  a  logical  chain  of  evidence,  and  making 
conceptual/theoretical  coherence.  The  researcher  determined  patterns  from  the  narrative 
of  the  two  focus  groups  and  verified  or  discounted  the  patterns  from  the  text  using  the 
voting  preferences  of  the  groups. 

Investigative  Question  One 

The  first  investigative  question  posed  in  Chapter  I  was,  “what  functions  the  system 
should  offer  the  user  in  terms  of  the  work  environment  and/or  the  desktop  environment?” 
There  were  several  general  functional  areas  the  discussion  centered  around  in  the  focus 
groups.  This  section  will  discuss  these  areas  in  detail  except  for  the  users’  desire  for  a 
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search  capability.  The  discussion  of  the  search  capability  will  be  performed  in  the  next 
section  concerning  investigative  question  two. 

The  first  topic  discussed  by  group  one  was  the  accessibility  of  the  system  to  the 
users.  The  second  group  also  determined  accessibility  as  a  key  requirement  for  the 
system.  In  addition  to  the  system  accessibility,  the  second  group  added  the  requirement 
for  a  backup  system  to  the  accessibility  idea.  Group  one  supported  the  requirement  for 
system  accessibility  with  62  votes  as  shown  in  Figure  15.  Group  two  didn’t  discuss  this 
topic  first,  and  didn’t  provide  as  much  support  through  the  voting  as  Figure  19  shows  the 
topic  of  broad  access  with  only  ten  votes. 

Group  one  discussed  a  set  of  functions  that  would  be  requirements  due  to  the 
system  being  on-line.  The  functions  include  central  update  of  changes  by  the  OPR, 
routing  of  annual  and  special  updates  performed  on-line,  and  a  link  to  the  OPR  for 
questions.  Group  one  supported  these  items  with  31, 16,  and  13  votes,  respectively  as 
shown  in  Figure  15.  Group  two  had  one  similar  suggestion  for  access  to  OPR  with  17 
votes  assigned  to  it. 

Both  groups  discussed  the  possibility  of  the  hyperlink  function  for  the  proposed 
system.  Group  one  put  20  votes  for  hyperlink  of  supplements,  tables,  forms,  and  other 
related  materials  as  shown  by  Figure  14.  Group  two  dedicated  only  five  votes  for  a 
general  hyperlink  requirement.  The  number  of  votes  for  the  hyperlink  function  didn’t 
support  the  pattern  established  by  the  group  discussions. 
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The  highest  vote  total  for  a  requirement  for  group  two  was  65  votes  for  system 
reliability.  The  other  group  didn’t  mention  the  topic  as  a  requirement.  The  strength  of 
conviction  by  group  one  supports  the  function  as  a  valid  requirement  for  the  system. 

The  requirements  recommended  from  task  three  in  the  procedure  include  the  items 
introduced  by  both  groups,  and  the  ideas  strongly  supported  by  only  one  group.  The 
clusters  in  Figures  22  and  23  show  the  requirements  both  groups  supported  and  the 
strength  of  the  support  in  votes.  Accessibility  mentioned  by  both  groups  and  strongly 
supported  by  group  one.  The  hyperlink  cluster  shows  both  groups  support  the  idea,  but 
with  only  20  and  5  votes.  The  other  cluster  with  both  groups  supporting  a  topic  is  the 
requirement  for  a  help  feature,  but  this  wasn’t  well  supported  with  only  six  and  two  votes. 
Some  requirements  received  support  from  only  one  group,  but  the  level  of  support  from 
the  single  group  suggests  the  feature  is  a  requirement.  System  reliability  received  65  votes 
from  group  two,  automatic  update  of  the  publications  received  31  votes  from  group  one, 
deployment  capability  received  20  votes  from  group  one,  and  access  to  the  OPR  received 
17  votes  from  group  two.  Table  four  summarizes  the  requirements  generated  from  task 
three  of  the  focus  group  procedure. 


Table  4.  Task  Three  Requirements 


Requirements  From  Both  Groups 

Requirements  From  One  Group 

Broad  system  accessibility 

Hyperlink  capability 

On-line  help 

System  reliability 

Changes  automatically  updated 
Deployment  capability 

On-line  access  to  OPR 
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Investigative  Question  Two 


This  section  draws  conclusions  on  the  groups’  ideas  concerning  the  second 
investigative  question,  “on  which  types  of  items  should  the  user  be  able  to  query  the 
publication  system:  i.e.  key  terms,  subject  areas  or  other  ideas?”  Both  focus  groups 
determined  a  requirement  for  a  search  capability  during  the  discussion  of  task  three.  The 
discussion  of  the  specific  requirements  of  the  search  capabihty  were  held  for  task  four. 

There  were  several  strongly  supported  clusters,  or  items  mentioned  by  both 
groups,  in  the  combined  group  bins  as  shown  in  Figures  24  and  25.  The  first  and  most 
heavily  supported  was  the  requirement  for  the  ability  to  search  the  text  of  the  documents 
for  a  particular  word.  The  groups  supported  the  word  search  of  the  text  with  65  and  23 
votes.  The  next  cluster  supported  the  requirement  for  searching  the  index  of  the 
publications  with  60  and  23  votes.  The  next  most  supported  cluster  gave  32  and  10  votes 
to  the  ability  to  use  delimiters  to  narrow  the  scope  of  the  search.  The  cluster  under  the 
mental  effort  bin  in  Figure  24  for  a  search  history  was  supported  with  22  and  5  votes.  The 
lowest  support  for  a  cluster  was  twelve  and  zero  votes  for  the  requirement  for  a  hit  count. 
The  ideas  receiving  significant  votes  from  only  one  group  included:  3 1  votes  to  have  the 
system  automatically  put  the  citation  on  text  cut  and  pasted  to  another  program,  22  votes 
for  the  ability  to  use  a  natural  language  or  advanced  search,  21  votes  for  the  ability  to 
mark  or  print  sections  of  the  text,  and  20  votes  to  the  ability  to  query  who  is  the  OPR  for 
a  publication.  Table  five  summarized  the  requirements  generated  from  task  four  of  the 
focus  group  procedure. 
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Table  5.  Task  Four  Requirements 


Requirements  From  Both  Groups 

Requirements  From  One  Group 

Word  search  of  document  text 

Search  index  of  publications 

Use  of  delimiters  to  narrow  search 
Maintain  a  search  history 

Provide  a  hit  count  for  word  searches 

Automatic  citations 

Natural  language  search 

Mark  or  print  sections  of  text 

Query  for  OPR 

Investigative  Question  Three 

The  final  investigative  question  posed  the  question,  “what  should  the  graphical 
interface  for  the  on-line  publications  system  look  like?”  Task  five  had  the  groups  draw 
representations  of  the  screens  the  system  would  use  as  the  interface.  Representations  of 
the  pictures  drawn  by  groups  two  and  three  are  included  in  Chapter  IV.  This  task  will 
draw  conclusions  from  the  groups’  drawings  from  task  five  and  from  the  discussion  areas 
relevant  to  the  subject  of  the  interface. 

Group  one  discussed  the  desire  for  the  system  to  look  like  Windows  or  an  Internet 
browser  the  users  are  familiar  with  from  their  work.  They  felt  the  need  to  learn  a  totally 
new  interface  would  be  a  waste  of  time.  Both  groups’  drawings  supported  this  statement. 
The  drawings  support  the  assumption  the  proposed  system  should  be  based  on  the 
graphical  interface  common  to  the  personal  computer  interfaces  used  throughout  the  Air 
Force. 

Limitations 

The  researcher  was  not  an  expert  facilitator  in  the  focus  group  process.  The  use  of 
the  pilot  study  helped  offset  this  limitation  as  it  gave  the  researcher  experience  and 
allowed  the  fine-tuning  of  the  procedure  for  the  remaining  groups.  The  focus  group 


75 


methodology  has  advantages  and  disadvantages  over  participant  observation  and 
individual  interviews,  which  are  the  data  gathering  techniques  for  which  the  focus  group 
developed  as  an  intermediate  method.  The  focus  group  allows  more  data  to  be  gathered 
in  a  shorter  amount  of  time  than  the  participant  method,  but  moves  the  research  to  an 
artificial  environment.  This  study  is  qualitative,  concerning  a  proposed  system,  and  the 
natural  environment  would  not  provide  any  different  results  for  this  research.  The 
individual  interview  has  an  advantage  as  the  researcher  controls  the  flow  of  the  interview 
and  can  organize  the  data  for  easier  analysis.  The  researcher  sought  the  interaction  of  the 
group  members  to  allow  the  teamwork  and  creativity  of  the  users  to  determine  the 
requirements.  In  addition  to  the  group  interaction  benefit,  the  researcher  counted  on 
Miles  andHuberman’s  qualitative  research  model  to  help  organize  and  draw  conclusions 
about  the  data. 

Recommendations  For  Future  Research 

During  the  course  of  the  research  process,  the  researcher  identified  two  areas  for 
future  research  that  could  lead  to  useful  results.  The  researcher  discussed  in  Chapter  I  the 
premise  that  providing  publications  on-line  could  provide  productivity  gains.  The  answer 
to  this  question  could  have  broad  impact  on  the  push  to  move  systems  to  the  on-line 
mode.  A  relevant  measure  of  productivity  could  help  the  cost  justification  effort  for 
supporting  the  move  to  on-line  systems. 

Another  research  effort  related  to  this  study  could  entail  using  the  results  from  this 
effort  to  produce  and  test  a  prototype  system.  The  systems  development  life  cycle 
supports  prototype  building  as  a  next  step  in  the  process  of  developing  the  proposed  on- 
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line  publishing  system.  The  model  of  the  SDLC  presented  by  Martin  (1995)  supports  this 
as  a  next  step  as  stage  four  in  the  model  is  system  selection  and  phase  five  is  preliminary 
design  and  system  study. 
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Appendix  A 
Project  Scope 

The  intention  of  the  group  session  is  to  consider  the  possible  look  and  feel  of  the 
on-line  publications  system  of  the  future  for  the  USAF.  WeTl  generate  ideas  concerning 
the  user  interface  and  the  possible  functionality  the  system  should  have.  We  don’t  expect 
to  come  up  the  exact  specifications  for  the  entire  system  in  the  one  and  one  half  hour 
session.  We  can  offer  the  system  planners  valuable  input  about  the  function,  look,  and  feel 
of  the  proposed  publishing  system.  The  planners  at  HQ  for  publications  have  had  little 
input  from  ±e  users  concerning  the  creation  of  an  on-line  publishing  system.  Input  from 
the  users  should  help  foster  user  acceptance  of  the  system  as  well  as  ensure  the  system 
offers  the  ability  to  meet  the  work  needs  they  have. 

Group  Function: 

Conceptualize  an  on-line  publishing  system  for  the  USAF. 

Tasks: 

1.  Complete  personal  information  worksheet  (3  min). 

2.  Creativity  worksheet  (5  min). 

3.  Consider  and  develop  a  list  of  the  functions  the  system  should  offer  the  user  in  the 
work  area  and/or  desktop  (30  min). 

4.  Describe  the  types  of  items  the  user  should  be  able  to  query  the  publication  system: 
i.e.  key  terms,  subject  areas  or  o±er  ideas  (10  min). 

5.  Create  a  visual  representation  of  the  screen  graphics,  buttons  and  colors.  Materials 
provided  will  include  a  flip  chart  and  various  colored  markers  (30  min). 

6.  Rank  Order  the  Suggestions  from  task  3  and  4  (10  min). 
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Appendix  B 

Personal  Information  Worksheet 

Gender  □  □ 

Male  Female 

Military/Civilian  Rank: _ ^ 

Please  enter  your  desired  response  to  the  following  statements  based  on  this  scale: 

Disagree  Agree 

< - > 

1  2  3  4  5  6  7 

I  consider  myself  experienced  with  the  general  use  of  computers.  _ 

I’ve  experienced  many  different  types  of  computer  interfaces.  _ 

I’m  an  experienced  user  of  US  AF  publications.  _ 
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I.Q.  TEST 


Hare  are  some  real  puzrlers  for  you!  Decipher  the 
hiddeu  meaning  of  each  se:  of  words. 


belt 

hitting 


EXIT 


onen 


often 

often 


night  fiy 


SPRING 

SUMMER 


migrAine  9 ALLS  If” 

,  WINTER 


a 

time 


GIVE 

GIVE 

GIVE 

GIVE 


GET 

GET 

GET 

GET 


EMPLOY  T 
MEN 


esroh  riding 


UNWASHED 


UNCLEAN 

SUUJED 


SMUDGED 

BESMIRCXS3 


XSflNISHED 


D012*'0R 
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Answers 


1.  Hitting  below  the  belt 

2.  Out  on  a  limb 

3.  More  often  than  not 

4.  Fly  by  night 

5.  A  splitting  headache 

6.  A  man  for  all  seasons 

7.  All  in  a  day’s  work 

8.  Westside  story 

9.  Forgive  and  forget 

10.  Short  of  breath 

11.  Accident  prone 

12.  Horseback  riding 

13.  Men  out  of  work 

14.  Just  under  the  wire 

15.  Dirty  dozen 

16.  A  foot  in  the  door 
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